climateprediction.net home page
Task 22291329

Task 22291329

Name oifs_43r3_ps_0174_2019050100_123_988_12204818_0
Workunit 12204818
Created 24 Dec 2022, 22:35:08 UTC
Sent 20 Jan 2023, 16:21:15 UTC
Report deadline 21 Mar 2023, 16:21:15 UTC
Received 21 Jan 2023, 13:50:15 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 17 hours 34 min 16 sec
CPU time 15 hours 59 min 40 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,408.81 MB
Peak swap size 4,978.08 MB
Peak disk usage 4,450.54 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
ar/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_109.zip
Uploading the intermediate file: upload_file_109.zip
  04:23:18 STEP 2641 H=2641:00 +CPU= 15.863
  04:23:34 STEP 2642 H=2642:00 +CPU= 15.830
  04:24:01 STEP 2643 H=2643:00 +CPU= 26.411
  04:24:16 STEP 2644 H=2644:00 +CPU= 15.842
  04:24:32 STEP 2645 H=2645:00 +CPU= 15.915
  04:24:58 STEP 2646 H=2646:00 +CPU= 25.834
  04:25:16 STEP 2647 H=2647:00 +CPU= 17.707
  04:25:34 STEP 2648 H=2648:00 +CPU= 17.908
  04:26:00 STEP 2649 H=2649:00 +CPU= 26.054
  04:26:17 STEP 2650 H=2650:00 +CPU= 16.657
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0174
(argv4) batchid: 988
(argv5) wuid: 12204818
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2005261802337c492d95763b471432f to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0174_2019050100_123_988_12204818.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0174_2019050100_123_988_12204818.zip
ic_ancil_file: ic_ancil_12204818
ifsdata_file: ifsdata_12204818
climate_data_file: clim_data_12204818
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_420a8cdb0adb277a5c06638f72bcbf1d to: /var/lib/boinc-client/slots/3/ic_ancil_12204818.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12204818.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9a2db819c383043e30fd45a0c39ac383 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12204818.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12204818.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12204818.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12204818.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12204818.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12204818.xml
-- Model is restarting --
Adjusting last_iter, 2651, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12204818.xml
last_cpu_time: 49934
upload_file_number: 110
last_iter: 2639
last_upload: 9504000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964
The child process has been launched with process id: 926575
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:926575:926575] [20230121:042723:1674300443:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:27:53 STEP 2640 H=2640:00 +CPU= 28.097
  04:28:09 STEP 2641 H=2641:00 +CPU= 15.250
  04:28:26 STEP 2642 H=2642:00 +CPU= 16.800
  04:28:53 STEP 2643 H=2643:00 +CPU= 26.855
  04:29:09 STEP 2644 H=2644:00 +CPU= 15.840
  04:29:25 STEP 2645 H=2645:00 +CPU= 16.166
  04:29:51 STEP 2646 H=2646:00 +CPU= 25.614
  04:30:08 STEP 2647 H=2647:00 +CPU= 17.018
  04:30:24 STEP 2648 H=2648:00 +CPU= 15.680
  04:30:51 STEP 2649 H=2649:00 +CPU= 26.549
  04:31:08 STEP 2650 H=2650:00 +CPU= 16.953
  04:31:24 STEP 2651 H=2651:00 +CPU= 16.771
  04:31:52 STEP 2652 H=2652:00 +CPU= 27.650
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002652
  04:32:09 STEP 2653 H=2653:00 +CPU= 16.736
  04:32:26 STEP 2654 H=2654:00 +CPU= 17.044
  04:32:52 STEP 2655 H=2655:00 +CPU= 25.162
  04:33:07 STEP 2656 H=2656:00 +CPU= 15.641
  04:33:24 STEP 2657 H=2657:00 +CPU= 16.509
  04:33:49 STEP 2658 H=2658:00 +CPU= 24.829
  04:34:07 STEP 2659 H=2659:00 +CPU= 17.531
  04:34:24 STEP 2660 H=2660:00 +CPU= 16.766
  04:34:49 STEP 2661 H=2661:00 +CPU= 25.189
  04:35:06 STEP 2662 H=2662:00 +CPU= 16.235
  04:35:22 STEP 2663 H=2663:00 +CPU= 16.272
  04:35:52 STEP 2664 H=2664:00 +CPU= 30.034
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_110.zip
Uploading the intermediate file: upload_file_110.zip
  04:36:08 STEP 2665 H=2665:00 +CPU= 15.534
  04:36:25 STEP 2666 H=2666:00 +CPU= 16.347
  04:36:50 STEP 2667 H=2667:00 +CPU= 25.366
  04:37:07 STEP 2668 H=2668:00 +CPU= 16.635
  04:37:23 STEP 2669 H=2669:00 +CPU= 15.576
  04:37:48 STEP 2670 H=2670:00 +CPU= 25.140
  04:38:04 STEP 2671 H=2671:00 +CPU= 15.732
  04:38:21 STEP 2672 H=2672:00 +CPU= 17.161
  04:38:46 STEP 2673 H=2673:00 +CPU= 25.149
  04:39:04 STEP 2674 H=2674:00 +CPU= 17.347
  04:39:21 STEP 2675 H=2675:00 +CPU= 16.710
  04:39:49 STEP 2676 H=2676:00 +CPU= 28.139
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002676
  04:40:06 STEP 2677 H=2677:00 +CPU= 16.547
  04:40:22 STEP 2678 H=2678:00 +CPU= 16.341
  04:40:46 STEP 2679 H=2679:00 +CPU= 23.318
  04:41:02 STEP 2680 H=2680:00 +CPU= 15.879
  04:41:19 STEP 2681 H=2681:00 +CPU= 16.830
  04:41:47 STEP 2682 H=2682:00 +CPU= 27.680
  04:42:04 STEP 2683 H=2683:00 +CPU= 16.993
  04:42:21 STEP 2684 H=2684:00 +CPU= 16.740
  04:42:47 STEP 2685 H=2685:00 +CPU= 26.168
  04:43:06 STEP 2686 H=2686:00 +CPU= 17.911
  04:43:24 STEP 2687 H=2687:00 +CPU= 18.143
  04:43:53 STEP 2688 H=2688:00 +CPU= 28.945
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_111.zip
Uploading the intermediate file: upload_file_111.zip
  04:44:10 STEP 2689 H=2689:00 +CPU= 16.750
  04:44:30 STEP 2690 H=2690:00 +CPU= 19.516
  04:44:57 STEP 2691 H=2691:00 +CPU= 26.735
  04:45:14 STEP 2692 H=2692:00 +CPU= 16.776
  04:45:32 STEP 2693 H=2693:00 +CPU= 17.820
  04:46:03 STEP 2694 H=2694:00 +CPU= 30.511
  04:46:21 STEP 2695 H=2695:00 +CPU= 17.821
  04:46:36 STEP 2696 H=2696:00 +CPU= 15.018
  04:47:03 STEP 2697 H=2697:00 +CPU= 26.666
  04:47:22 STEP 2698 H=2698:00 +CPU= 18.721
  04:47:41 STEP 2699 H=2699:00 +CPU= 18.989
  04:48:12 STEP 2700 H=2700:00 +CPU= 30.042
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002700
  04:48:33 STEP 2701 H=2701:00 +CPU= 20.769
  04:48:52 STEP 2702 H=2702:00 +CPU= 19.146
  04:49:22 STEP 2703 H=2703:00 +CPU= 29.340
  04:49:41 STEP 2704 H=2704:00 +CPU= 19.064
  04:50:00 STEP 2705 H=2705:00 +CPU= 18.773
  04:50:28 STEP 2706 H=2706:00 +CPU= 27.443
  04:50:47 STEP 2707 H=2707:00 +CPU= 18.616
  04:51:08 STEP 2708 H=2708:00 +CPU= 21.019
  04:51:38 STEP 2709 H=2709:00 +CPU= 29.424
  04:51:58 STEP 2710 H=2710:00 +CPU= 19.168
  04:52:17 STEP 2711 H=2711:00 +CPU= 19.135
  04:52:50 STEP 2712 H=2712:00 +CPU= 32.686
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_112.zip
Uploading the intermediate file: upload_file_112.zip
  04:53:09 STEP 2713 H=2713:00 +CPU= 18.275
  04:53:29 STEP 2714 H=2714:00 +CPU= 19.410
  04:53:57 STEP 2715 H=2715:00 +CPU= 28.147
  04:54:15 STEP 2716 H=2716:00 +CPU= 17.645
  04:54:33 STEP 2717 H=2717:00 +CPU= 18.060
  04:55:02 STEP 2718 H=2718:00 +CPU= 28.544
  04:55:20 STEP 2719 H=2719:00 +CPU= 17.347
  04:55:40 STEP 2720 H=2720:00 +CPU= 19.842
  04:56:09 STEP 2721 H=2721:00 +CPU= 29.016
  04:56:28 STEP 2722 H=2722:00 +CPU= 18.113
  04:56:46 STEP 2723 H=2723:00 +CPU= 17.737
  04:57:18 STEP 2724 H=2724:00 +CPU= 31.707
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002724
  04:57:38 STEP 2725 H=2725:00 +CPU= 20.119
  04:57:58 STEP 2726 H=2726:00 +CPU= 19.701
  04:58:26 STEP 2727 H=2727:00 +CPU= 27.012
  04:58:45 STEP 2728 H=2728:00 +CPU= 18.959
  04:59:05 STEP 2729 H=2729:00 +CPU= 19.560
  04:59:33 STEP 2730 H=2730:00 +CPU= 28.077
  04:59:53 STEP 2731 H=2731:00 +CPU= 19.546
  05:00:11 STEP 2732 H=2732:00 +CPU= 17.844
  05:00:41 STEP 2733 H=2733:00 +CPU= 29.400
  05:00:59 STEP 2734 H=2734:00 +CPU= 17.386
  05:01:18 STEP 2735 H=2735:00 +CPU= 18.739
  05:01:53 STEP 2736 H=2736:00 +CPU= 34.693
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_113.zip
Uploading the intermediate file: upload_file_113.zip
  05:02:12 STEP 2737 H=2737:00 +CPU= 18.987
  05:02:33 STEP 2738 H=2738:00 +CPU= 19.778
  05:03:03 STEP 2739 H=2739:00 +CPU= 29.597
  05:03:21 STEP 2740 H=2740:00 +CPU= 18.018
  05:03:38 STEP 2741 H=2741:00 +CPU= 17.038
  05:04:06 STEP 2742 H=2742:00 +CPU= 28.113
  05:04:25 STEP 2743 H=2743:00 +CPU= 18.590
  05:04:43 STEP 2744 H=2744:00 +CPU= 18.004
  05:05:11 STEP 2745 H=2745:00 +CPU= 27.030
  05:05:28 STEP 2746 H=2746:00 +CPU= 17.601
  05:05:47 STEP 2747 H=2747:00 +CPU= 18.553
  05:06:18 STEP 2748 H=2748:00 +CPU= 30.097
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002748
  05:06:36 STEP 2749 H=2749:00 +CPU= 18.085
  05:06:54 STEP 2750 H=2750:00 +CPU= 17.143
  05:07:23 STEP 2751 H=2751:00 +CPU= 28.779
  05:07:42 STEP 2752 H=2752:00 +CPU= 19.089
  05:08:00 STEP 2753 H=2753:00 +CPU= 17.816
  05:08:26 STEP 2754 H=2754:00 +CPU= 25.849
  05:08:46 STEP 2755 H=2755:00 +CPU= 19.097
  05:09:06 STEP 2756 H=2756:00 +CPU= 19.522
  05:09:36 STEP 2757 H=2757:00 +CPU= 29.766
  05:09:57 STEP 2758 H=2758:00 +CPU= 20.879
  05:10:18 STEP 2759 H=2759:00 +CPU= 20.493
  05:10:52 STEP 2760 H=2760:00 +CPU= 33.051
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_114.zip
Uploading the intermediate file: upload_file_114.zip
  05:11:11 STEP 2761 H=2761:00 +CPU= 18.826
  05:11:28 STEP 2762 H=2762:00 +CPU= 16.931
  05:11:55 STEP 2763 H=2763:00 +CPU= 26.557
  05:12:14 STEP 2764 H=2764:00 +CPU= 18.615
  05:12:33 STEP 2765 H=2765:00 +CPU= 18.414
  05:13:02 STEP 2766 H=2766:00 +CPU= 29.319
  05:13:22 STEP 2767 H=2767:00 +CPU= 19.040
  05:13:41 STEP 2768 H=2768:00 +CPU= 18.692
  05:14:10 STEP 2769 H=2769:00 +CPU= 28.585
  05:14:29 STEP 2770 H=2770:00 +CPU= 19.020
  05:14:48 STEP 2771 H=2771:00 +CPU= 18.398
  05:15:17 STEP 2772 H=2772:00 +CPU= 29.150
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002772
  05:15:36 STEP 2773 H=2773:00 +CPU= 18.473
  05:15:55 STEP 2774 H=2774:00 +CPU= 19.397
  05:16:26 STEP 2775 H=2775:00 +CPU= 29.996
  05:16:46 STEP 2776 H=2776:00 +CPU= 19.724
  05:17:06 STEP 2777 H=2777:00 +CPU= 19.807
  05:17:35 STEP 2778 H=2778:00 +CPU= 28.350
  05:17:53 STEP 2779 H=2779:00 +CPU= 17.508
  05:18:12 STEP 2780 H=2780:00 +CPU= 18.733
  05:18:41 STEP 2781 H=2781:00 +CPU= 28.578
  05:18:59 STEP 2782 H=2782:00 +CPU= 17.651
  05:19:17 STEP 2783 H=2783:00 +CPU= 17.762
  05:19:50 STEP 2784 H=2784:00 +CPU= 32.415
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_115.zip
Uploading the intermediate file: upload_file_115.zip
  05:20:10 STEP 2785 H=2785:00 +CPU= 19.721
  05:20:28 STEP 2786 H=2786:00 +CPU= 18.017
  05:20:56 STEP 2787 H=2787:00 +CPU= 27.689
  05:21:14 STEP 2788 H=2788:00 +CPU= 18.059
  05:21:34 STEP 2789 H=2789:00 +CPU= 19.259
  05:22:04 STEP 2790 H=2790:00 +CPU= 30.150
  05:22:22 STEP 2791 H=2791:00 +CPU= 17.948
  05:22:40 STEP 2792 H=2792:00 +CPU= 17.773
  05:23:08 STEP 2793 H=2793:00 +CPU= 27.417
  05:23:28 STEP 2794 H=2794:00 +CPU= 19.126
  05:23:47 STEP 2795 H=2795:00 +CPU= 19.420
  05:24:20 STEP 2796 H=2796:00 +CPU= 32.847
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002796
  05:24:40 STEP 2797 H=2797:00 +CPU= 19.448
  05:25:00 STEP 2798 H=2798:00 +CPU= 19.940
  05:25:28 STEP 2799 H=2799:00 +CPU= 27.156
  05:25:47 STEP 2800 H=2800:00 +CPU= 18.856
  05:26:06 STEP 2801 H=2801:00 +CPU= 19.093
  05:26:37 STEP 2802 H=2802:00 +CPU= 30.457
  05:26:54 STEP 2803 H=2803:00 +CPU= 17.109
  05:27:14 STEP 2804 H=2804:00 +CPU= 19.923
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0174
(argv4) batchid: 988
(argv5) wuid: 12204818
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2005261802337c492d95763b471432f to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0174_2019050100_123_988_12204818.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0174_2019050100_123_988_12204818.zip
ic_ancil_file: ic_ancil_12204818
ifsdata_file: ifsdata_12204818
climate_data_file: clim_data_12204818
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_420a8cdb0adb277a5c06638f72bcbf1d to: /var/lib/boinc-client/slots/3/ic_ancil_12204818.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12204818.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9a2db819c383043e30fd45a0c39ac383 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12204818.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12204818.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12204818.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12204818.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12204818.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12204818.xml
-- Model is restarting --
Adjusting last_iter, 2805, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12204818.xml
last_cpu_time: 53489
upload_file_number: 116
last_iter: 2783
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964
The child process has been launched with process id: 936780
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:936780:936780] [20230121:052830:1674304110:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:29:06 STEP 2784 H=2784:00 +CPU= 32.160
  05:29:25 STEP 2785 H=2785:00 +CPU= 18.787
  05:29:45 STEP 2786 H=2786:00 +CPU= 20.149
  05:30:15 STEP 2787 H=2787:00 +CPU= 29.512
  05:30:35 STEP 2788 H=2788:00 +CPU= 19.663
  05:30:52 STEP 2789 H=2789:00 +CPU= 17.082
  05:31:19 STEP 2790 H=2790:00 +CPU= 27.022
  05:31:38 STEP 2791 H=2791:00 +CPU= 18.418
  05:31:57 STEP 2792 H=2792:00 +CPU= 18.849
  05:32:24 STEP 2793 H=2793:00 +CPU= 26.576
  05:32:43 STEP 2794 H=2794:00 +CPU= 18.409
  05:33:02 STEP 2795 H=2795:00 +CPU= 18.675
  05:33:32 STEP 2796 H=2796:00 +CPU= 29.599
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002796
  05:33:51 STEP 2797 H=2797:00 +CPU= 18.789
  05:34:12 STEP 2798 H=2798:00 +CPU= 21.104
  05:34:41 STEP 2799 H=2799:00 +CPU= 27.995
  05:34:59 STEP 2800 H=2800:00 +CPU= 18.156
  05:35:16 STEP 2801 H=2801:00 +CPU= 16.969
  05:35:44 STEP 2802 H=2802:00 +CPU= 27.695
  05:36:03 STEP 2803 H=2803:00 +CPU= 18.547
  05:36:23 STEP 2804 H=2804:00 +CPU= 19.474
  05:36:51 STEP 2805 H=2805:00 +CPU= 27.008
  05:37:10 STEP 2806 H=2806:00 +CPU= 19.420
  05:37:31 STEP 2807 H=2807:00 +CPU= 20.412
  05:38:01 STEP 2808 H=2808:00 +CPU= 29.407
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_116.zip
Uploading the intermediate file: upload_file_116.zip
  05:38:19 STEP 2809 H=2809:00 +CPU= 17.382
  05:38:40 STEP 2810 H=2810:00 +CPU= 20.411
  05:39:09 STEP 2811 H=2811:00 +CPU= 29.087
  05:39:28 STEP 2812 H=2812:00 +CPU= 18.756
  05:39:46 STEP 2813 H=2813:00 +CPU= 18.075
  05:40:15 STEP 2814 H=2814:00 +CPU= 28.534
  05:40:34 STEP 2815 H=2815:00 +CPU= 18.651
  05:40:52 STEP 2816 H=2816:00 +CPU= 17.369
  05:41:19 STEP 2817 H=2817:00 +CPU= 26.945
  05:41:38 STEP 2818 H=2818:00 +CPU= 19.151
  05:41:58 STEP 2819 H=2819:00 +CPU= 19.796
  05:42:28 STEP 2820 H=2820:00 +CPU= 29.002
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002820
  05:42:46 STEP 2821 H=2821:00 +CPU= 18.083
  05:43:05 STEP 2822 H=2822:00 +CPU= 18.425
  05:43:32 STEP 2823 H=2823:00 +CPU= 27.461
  05:43:51 STEP 2824 H=2824:00 +CPU= 18.498
  05:44:08 STEP 2825 H=2825:00 +CPU= 17.129
  05:44:37 STEP 2826 H=2826:00 +CPU= 28.245
  05:44:57 STEP 2827 H=2827:00 +CPU= 19.219
  05:45:18 STEP 2828 H=2828:00 +CPU= 20.370
  05:45:45 STEP 2829 H=2829:00 +CPU= 27.258
  05:46:05 STEP 2830 H=2830:00 +CPU= 19.184
  05:46:25 STEP 2831 H=2831:00 +CPU= 20.007
  05:46:58 STEP 2832 H=2832:00 +CPU= 32.525
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_117.zip
Uploading the intermediate file: upload_file_117.zip
  05:47:15 STEP 2833 H=2833:00 +CPU= 16.245
  05:47:34 STEP 2834 H=2834:00 +CPU= 19.115
  05:48:04 STEP 2835 H=2835:00 +CPU= 29.146
  05:48:21 STEP 2836 H=2836:00 +CPU= 17.432
  05:48:39 STEP 2837 H=2837:00 +CPU= 17.892
  05:49:07 STEP 2838 H=2838:00 +CPU= 27.928
  05:49:25 STEP 2839 H=2839:00 +CPU= 17.226
  05:49:45 STEP 2840 H=2840:00 +CPU= 19.439
  05:50:11 STEP 2841 H=2841:00 +CPU= 26.439
  05:50:29 STEP 2842 H=2842:00 +CPU= 17.779
  05:50:48 STEP 2843 H=2843:00 +CPU= 18.103
  05:51:17 STEP 2844 H=2844:00 +CPU= 29.447
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002844
  05:51:36 STEP 2845 H=2845:00 +CPU= 18.222
  05:51:55 STEP 2846 H=2846:00 +CPU= 18.953
  05:52:24 STEP 2847 H=2847:00 +CPU= 27.854
  05:52:42 STEP 2848 H=2848:00 +CPU= 18.675
  05:52:59 STEP 2849 H=2849:00 +CPU= 16.296
  05:53:28 STEP 2850 H=2850:00 +CPU= 28.253
  05:53:46 STEP 2851 H=2851:00 +CPU= 17.955
  05:54:06 STEP 2852 H=2852:00 +CPU= 19.532
  05:54:32 STEP 2853 H=2853:00 +CPU= 26.267
  05:54:50 STEP 2854 H=2854:00 +CPU= 17.782
  05:55:11 STEP 2855 H=2855:00 +CPU= 20.674
  05:55:43 STEP 2856 H=2856:00 +CPU= 31.403
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_118.zip
Uploading the intermediate file: upload_file_118.zip
  05:55:59 STEP 2857 H=2857:00 +CPU= 15.110
  05:56:16 STEP 2858 H=2858:00 +CPU= 17.626
  05:56:46 STEP 2859 H=2859:00 +CPU= 28.596
  05:57:02 STEP 2860 H=2860:00 +CPU= 16.193
  05:57:19 STEP 2861 H=2861:00 +CPU= 16.981
  05:57:49 STEP 2862 H=2862:00 +CPU= 29.399
  05:58:08 STEP 2863 H=2863:00 +CPU= 18.582
  05:58:26 STEP 2864 H=2864:00 +CPU= 17.963
  05:58:54 STEP 2865 H=2865:00 +CPU= 28.319
  05:59:13 STEP 2866 H=2866:00 +CPU= 18.678
  05:59:31 STEP 2867 H=2867:00 +CPU= 17.040
  06:00:01 STEP 2868 H=2868:00 +CPU= 30.353
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002868
  06:00:21 STEP 2869 H=2869:00 +CPU= 19.775
  06:00:42 STEP 2870 H=2870:00 +CPU= 20.161
  06:01:10 STEP 2871 H=2871:00 +CPU= 28.102
  06:01:30 STEP 2872 H=2872:00 +CPU= 19.011
  06:01:49 STEP 2873 H=2873:00 +CPU= 18.712
  06:02:17 STEP 2874 H=2874:00 +CPU= 27.830
  06:02:35 STEP 2875 H=2875:00 +CPU= 18.040
  06:02:55 STEP 2876 H=2876:00 +CPU= 19.132
  06:03:22 STEP 2877 H=2877:00 +CPU= 26.403
  06:03:40 STEP 2878 H=2878:00 +CPU= 18.123
  06:03:59 STEP 2879 H=2879:00 +CPU= 18.987
  06:04:32 STEP 2880 H=2880:00 +CPU= 32.285
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_119.zip
Uploading the intermediate file: upload_file_119.zip
  06:04:50 STEP 2881 H=2881:00 +CPU= 17.136
  06:05:08 STEP 2882 H=2882:00 +CPU= 17.944
  06:05:35 STEP 2883 H=2883:00 +CPU= 27.199
  06:05:52 STEP 2884 H=2884:00 +CPU= 16.644
  06:06:10 STEP 2885 H=2885:00 +CPU= 17.204
  06:06:39 STEP 2886 H=2886:00 +CPU= 29.070
  06:06:58 STEP 2887 H=2887:00 +CPU= 18.428
  06:07:16 STEP 2888 H=2888:00 +CPU= 18.487
  06:07:43 STEP 2889 H=2889:00 +CPU= 26.730
  06:08:02 STEP 2890 H=2890:00 +CPU= 18.597
  06:08:18 STEP 2891 H=2891:00 +CPU= 15.878
  06:08:48 STEP 2892 H=2892:00 +CPU= 28.846
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002892
  06:09:07 STEP 2893 H=2893:00 +CPU= 19.518
  06:09:28 STEP 2894 H=2894:00 +CPU= 20.159
  06:09:56 STEP 2895 H=2895:00 +CPU= 28.053
  06:10:14 STEP 2896 H=2896:00 +CPU= 18.070
  06:10:33 STEP 2897 H=2897:00 +CPU= 18.404
  06:11:01 STEP 2898 H=2898:00 +CPU= 27.492
  06:11:19 STEP 2899 H=2899:00 +CPU= 17.349
  06:11:37 STEP 2900 H=2900:00 +CPU= 18.127
  06:12:06 STEP 2901 H=2901:00 +CPU= 28.606
  06:12:26 STEP 2902 H=2902:00 +CPU= 19.713
  06:12:45 STEP 2903 H=2903:00 +CPU= 18.585
  06:13:18 STEP 2904 H=2904:00 +CPU= 32.053
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_120.zip
Uploading the intermediate file: upload_file_120.zip
  06:13:36 STEP 2905 H=2905:00 +CPU= 17.873
  06:13:54 STEP 2906 H=2906:00 +CPU= 17.480
  06:14:20 STEP 2907 H=2907:00 +CPU= 25.790
  06:14:41 STEP 2908 H=2908:00 +CPU= 20.349
  06:14:58 STEP 2909 H=2909:00 +CPU= 17.592
  06:15:25 STEP 2910 H=2910:00 +CPU= 26.581
  06:15:43 STEP 2911 H=2911:00 +CPU= 17.555
  06:16:03 STEP 2912 H=2912:00 +CPU= 19.950
  06:16:31 STEP 2913 H=2913:00 +CPU= 27.794
  06:16:48 STEP 2914 H=2914:00 +CPU= 16.633
  06:17:06 STEP 2915 H=2915:00 +CPU= 18.191
  06:17:40 STEP 2916 H=2916:00 +CPU= 33.216
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002916
  06:17:57 STEP 2917 H=2917:00 +CPU= 16.759
  06:18:14 STEP 2918 H=2918:00 +CPU= 17.249
  06:18:42 STEP 2919 H=2919:00 +CPU= 26.847
  06:19:01 STEP 2920 H=2920:00 +CPU= 19.194
  06:19:20 STEP 2921 H=2921:00 +CPU= 18.972
  06:19:49 STEP 2922 H=2922:00 +CPU= 28.380
  06:20:06 STEP 2923 H=2923:00 +CPU= 16.571
  06:20:26 STEP 2924 H=2924:00 +CPU= 19.422
  06:20:55 STEP 2925 H=2925:00 +CPU= 28.775
  06:21:11 STEP 2926 H=2926:00 +CPU= 15.762
  06:21:29 STEP 2927 H=2927:00 +CPU= 17.165
  06:22:02 STEP 2928 H=2928:00 +CPU= 32.911
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_121.zip
Uploading the intermediate file: upload_file_121.zip
  06:22:20 STEP 2929 H=2929:00 +CPU= 17.322
  06:22:38 STEP 2930 H=2930:00 +CPU= 18.431
  06:23:06 STEP 2931 H=2931:00 +CPU= 27.851
  06:23:25 STEP 2932 H=2932:00 +CPU= 18.066
  06:23:40 STEP 2933 H=2933:00 +CPU= 15.408
  06:24:06 STEP 2934 H=2934:00 +CPU= 25.345
  06:24:25 STEP 2935 H=2935:00 +CPU= 18.873
  06:24:44 STEP 2936 H=2936:00 +CPU= 19.084
  06:25:12 STEP 2937 H=2937:00 +CPU= 27.096
  06:25:31 STEP 2938 H=2938:00 +CPU= 19.217
  06:25:51 STEP 2939 H=2939:00 +CPU= 19.468
  06:26:21 STEP 2940 H=2940:00 +CPU= 29.536
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002940
  06:26:37 STEP 2941 H=2941:00 +CPU= 16.628
  06:26:58 STEP 2942 H=2942:00 +CPU= 20.018
  06:27:28 STEP 2943 H=2943:00 +CPU= 29.853
  06:27:47 STEP 2944 H=2944:00 +CPU= 18.893
  06:28:06 STEP 2945 H=2945:00 +CPU= 18.602
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0174
(argv4) batchid: 988
(argv5) wuid: 12204818
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c2005261802337c492d95763b471432f to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0174_2019050100_123_988_12204818.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0174_2019050100_123_988_12204818.zip
ic_ancil_file: ic_ancil_12204818
ifsdata_file: ifsdata_12204818
climate_data_file: clim_data_12204818
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_420a8cdb0adb277a5c06638f72bcbf1d to: /var/lib/boinc-client/slots/3/ic_ancil_12204818.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12204818.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9a2db819c383043e30fd45a0c39ac383 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12204818.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12204818.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12204818.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12204818.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12204818.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12204818.xml
-- Model is restarting --
Adjusting last_iter, 2946, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12204818.xml
last_cpu_time: 57039
upload_file_number: 122
last_iter: 2927
last_upload: 10540800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964
The child process has been launched with process id: 946099
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:946099:946099] [20230121:062935:1674307775:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:30:09 STEP 2928 H=2928:00 +CPU= 30.570
  06:30:27 STEP 2929 H=2929:00 +CPU= 17.645
  06:30:44 STEP 2930 H=2930:00 +CPU= 17.880
  06:31:11 STEP 2931 H=2931:00 +CPU= 26.267
  06:31:29 STEP 2932 H=2932:00 +CPU= 18.150
  06:31:47 STEP 2933 H=2933:00 +CPU= 17.290
  06:32:14 STEP 2934 H=2934:00 +CPU= 26.448
  06:32:32 STEP 2935 H=2935:00 +CPU= 18.462
  06:32:51 STEP 2936 H=2936:00 +CPU= 17.943
  06:33:17 STEP 2937 H=2937:00 +CPU= 26.088
  06:33:36 STEP 2938 H=2938:00 +CPU= 18.663
  06:33:54 STEP 2939 H=2939:00 +CPU= 18.252
  06:34:25 STEP 2940 H=2940:00 +CPU= 29.841
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002940
  06:34:42 STEP 2941 H=2941:00 +CPU= 16.911
  06:35:00 STEP 2942 H=2942:00 +CPU= 17.638
  06:35:28 STEP 2943 H=2943:00 +CPU= 28.031
  06:35:46 STEP 2944 H=2944:00 +CPU= 17.770
  06:36:04 STEP 2945 H=2945:00 +CPU= 17.446
  06:36:31 STEP 2946 H=2946:00 +CPU= 26.914
  06:36:50 STEP 2947 H=2947:00 +CPU= 18.386
  06:37:07 STEP 2948 H=2948:00 +CPU= 17.353
  06:37:33 STEP 2949 H=2949:00 +CPU= 25.078
  06:37:52 STEP 2950 H=2950:00 +CPU= 18.674
  06:38:12 STEP 2951 H=2951:00 +CPU= 19.411
  06:38:44 STEP 2952 H=2952:00 +CPU= 31.476
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12204818/ICMGGhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0174_2019050100_123_988_12204818_0_r1586068964_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
06:42:07 (946096): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:33:18 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 10,623,600 57,580 0.0054
09 May 2023 16:32:58 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 8,380,800 43,850 0.0052
09 May 2023 16:32:53 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 7,516,800 39,631 0.0053
09 May 2023 16:32:46 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 6,652,800 35,382 0.0053
09 May 2023 16:32:34 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 5,788,800 31,144 0.0054
09 May 2023 16:32:29 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 4,924,800 26,906 0.0055
09 May 2023 16:32:23 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 4,060,800 22,667 0.0056
09 May 2023 16:32:17 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 3,196,800 18,419 0.0058
09 May 2023 16:32:12 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 2,332,800 14,190 0.0061
09 May 2023 16:31:59 1532237 22291329 oifs_43r3_ps_0174_2019050100_123_988_12204818_0 1,468,800 9,973 0.0068


©2024 climateprediction.net