climateprediction.net (CPDN) home page
Task 22303871

Task 22303871

Name oifs_43r3_ps_0897_2010050100_123_979_12196541_1
Workunit 12196541
Created 26 Jan 2023, 10:48:48 UTC
Sent 26 Jan 2023, 10:49:10 UTC
Report deadline 27 Mar 2023, 10:49:10 UTC
Received 28 Jan 2023, 8:56:19 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1493665
Run time 11 hours 22 min 40 sec
CPU time 10 hours 21 min 30 sec
Validate state Invalid
Credit 2,298.24
Device peak FLOPS 4.47 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4.38 GB
Peak swap size 4.86 GB
Peak disk usage 7.50 GB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22427:22427] [20230127:162221:1674861741:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:23:21 STEP  744 H= 744:00 +CPU= 48.510
  16:23:51 STEP  745 H= 745:00 +CPU= 27.323
  16:24:21 STEP  746 H= 746:00 +CPU= 27.302
  16:25:08 STEP  747 H= 747:00 +CPU= 42.644
  16:25:38 STEP  748 H= 748:00 +CPU= 27.423
  16:26:09 STEP  749 H= 749:00 +CPU= 27.241
  16:26:55 STEP  750 H= 750:00 +CPU= 42.271
  16:27:20 STEP  751 H= 751:00 +CPU= 23.749
  16:27:49 STEP  752 H= 752:00 +CPU= 24.784
  16:28:29 STEP  753 H= 753:00 +CPU= 37.655
  16:28:55 STEP  754 H= 754:00 +CPU= 24.513
  16:29:23 STEP  755 H= 755:00 +CPU= 23.987
  16:30:15 STEP  756 H= 756:00 +CPU= 42.495
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000756
  16:30:41 STEP  757 H= 757:00 +CPU= 23.740
  16:31:06 STEP  758 H= 758:00 +CPU= 24.592
  16:31:47 STEP  759 H= 759:00 +CPU= 37.558
  16:32:13 STEP  760 H= 760:00 +CPU= 24.673
  16:32:42 STEP  761 H= 761:00 +CPU= 24.213
  16:33:33 STEP  762 H= 762:00 +CPU= 40.475
  16:34:04 STEP  763 H= 763:00 +CPU= 24.727
  16:34:37 STEP  764 H= 764:00 +CPU= 25.579
  16:35:28 STEP  765 H= 765:00 +CPU= 39.527
  16:36:01 STEP  766 H= 766:00 +CPU= 25.601
  16:36:33 STEP  767 H= 767:00 +CPU= 25.446
  16:37:32 STEP  768 H= 768:00 +CPU= 45.754
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_31.zip
Uploading the intermediate file: upload_file_31.zip
  16:38:03 STEP  769 H= 769:00 +CPU= 24.741
  16:38:35 STEP  770 H= 770:00 +CPU= 25.583
  16:39:23 STEP  771 H= 771:00 +CPU= 39.345
  16:39:54 STEP  772 H= 772:00 +CPU= 25.525
  16:40:25 STEP  773 H= 773:00 +CPU= 24.772
  16:41:15 STEP  774 H= 774:00 +CPU= 40.695
  16:41:45 STEP  775 H= 775:00 +CPU= 24.952
  16:42:18 STEP  776 H= 776:00 +CPU= 25.885
  16:43:06 STEP  777 H= 777:00 +CPU= 39.877
  16:43:37 STEP  778 H= 778:00 +CPU= 25.665
  16:44:07 STEP  779 H= 779:00 +CPU= 24.843
  16:44:53 STEP  780 H= 780:00 +CPU= 42.912
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000780
  16:45:17 STEP  781 H= 781:00 +CPU= 24.078
  16:45:43 STEP  782 H= 782:00 +CPU= 24.933
  16:46:22 STEP  783 H= 783:00 +CPU= 38.497
  16:46:47 STEP  784 H= 784:00 +CPU= 24.754
  16:47:12 STEP  785 H= 785:00 +CPU= 23.957
  16:47:52 STEP  786 H= 786:00 +CPU= 39.340
  16:48:17 STEP  787 H= 787:00 +CPU= 24.449
  16:48:43 STEP  788 H= 788:00 +CPU= 25.454
  16:49:22 STEP  789 H= 789:00 +CPU= 39.202
  16:49:48 STEP  790 H= 790:00 +CPU= 25.558
  16:50:15 STEP  791 H= 791:00 +CPU= 25.719
  16:51:01 STEP  792 H= 792:00 +CPU= 45.672
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_32.zip
Uploading the intermediate file: upload_file_32.zip
  16:51:26 STEP  793 H= 793:00 +CPU= 23.993
  16:51:51 STEP  794 H= 794:00 +CPU= 25.363
  16:52:33 STEP  795 H= 795:00 +CPU= 38.763
  16:52:59 STEP  796 H= 796:00 +CPU= 25.144
  16:53:26 STEP  797 H= 797:00 +CPU= 24.671
  16:54:08 STEP  798 H= 798:00 +CPU= 39.809
  16:54:33 STEP  799 H= 799:00 +CPU= 24.260
  16:54:59 STEP  800 H= 800:00 +CPU= 25.266
  16:55:41 STEP  801 H= 801:00 +CPU= 39.761
  16:56:08 STEP  802 H= 802:00 +CPU= 25.244
  16:56:36 STEP  803 H= 803:00 +CPU= 24.891
  16:57:20 STEP  804 H= 804:00 +CPU= 43.302
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000804
  16:57:47 STEP  805 H= 805:00 +CPU= 24.637
  16:58:17 STEP  806 H= 806:00 +CPU= 25.725
  16:58:56 STEP  807 H= 807:00 +CPU= 38.547
  16:59:22 STEP  808 H= 808:00 +CPU= 25.175
  16:59:46 STEP  809 H= 809:00 +CPU= 24.023
  17:00:26 STEP  810 H= 810:00 +CPU= 39.429
  17:00:51 STEP  811 H= 811:00 +CPU= 24.203
  17:01:16 STEP  812 H= 812:00 +CPU= 24.896
  17:01:56 STEP  813 H= 813:00 +CPU= 38.710
  17:02:22 STEP  814 H= 814:00 +CPU= 25.323
  17:02:48 STEP  815 H= 815:00 +CPU= 25.195
  17:03:35 STEP  816 H= 816:00 +CPU= 45.667
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_33.zip
Uploading the intermediate file: upload_file_33.zip
  17:04:05 STEP  817 H= 817:00 +CPU= 24.846
  17:04:32 STEP  818 H= 818:00 +CPU= 25.681
  17:05:13 STEP  819 H= 819:00 +CPU= 39.613
  17:05:40 STEP  820 H= 820:00 +CPU= 25.684
  17:06:06 STEP  821 H= 821:00 +CPU= 24.351
  17:06:49 STEP  822 H= 822:00 +CPU= 40.394
  17:07:14 STEP  823 H= 823:00 +CPU= 24.918
  17:07:40 STEP  824 H= 824:00 +CPU= 25.285
  17:08:19 STEP  825 H= 825:00 +CPU= 38.309
  17:08:45 STEP  826 H= 826:00 +CPU= 25.138
  17:09:09 STEP  827 H= 827:00 +CPU= 24.164
  17:09:56 STEP  828 H= 828:00 +CPU= 43.263
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000828
  17:10:23 STEP  829 H= 829:00 +CPU= 24.595
  17:10:48 STEP  830 H= 830:00 +CPU= 25.077
  17:11:28 STEP  831 H= 831:00 +CPU= 38.876
  17:11:53 STEP  832 H= 832:00 +CPU= 25.138
  17:12:18 STEP  833 H= 833:00 +CPU= 24.458
  17:12:59 STEP  834 H= 834:00 +CPU= 40.426
  17:13:24 STEP  835 H= 835:00 +CPU= 24.215
  17:13:50 STEP  836 H= 836:00 +CPU= 25.247
  17:14:29 STEP  837 H= 837:00 +CPU= 38.445
  17:14:55 STEP  838 H= 838:00 +CPU= 25.022
  17:15:20 STEP  839 H= 839:00 +CPU= 25.344
  17:16:11 STEP  840 H= 840:00 +CPU= 44.901
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_34.zip
Uploading the intermediate file: upload_file_34.zip
  17:16:36 STEP  841 H= 841:00 +CPU= 23.981
  17:17:01 STEP  842 H= 842:00 +CPU= 24.746
  17:17:41 STEP  843 H= 843:00 +CPU= 38.591
  17:18:06 STEP  844 H= 844:00 +CPU= 25.215
  17:18:31 STEP  845 H= 845:00 +CPU= 24.390
  17:19:12 STEP  846 H= 846:00 +CPU= 39.716
  17:19:37 STEP  847 H= 847:00 +CPU= 24.400
  17:20:03 STEP  848 H= 848:00 +CPU= 25.475
  17:20:42 STEP  849 H= 849:00 +CPU= 38.426
  17:21:07 STEP  850 H= 850:00 +CPU= 24.848
  17:21:32 STEP  851 H= 851:00 +CPU= 24.217
  17:22:15 STEP  852 H= 852:00 +CPU= 42.593
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000852
  17:22:40 STEP  853 H= 853:00 +CPU= 24.236
  17:23:06 STEP  854 H= 854:00 +CPU= 25.078
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0897
(argv4) batchid: 979
(argv5) wuid: 12196541
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03dbe9c1a0aea012c6b90d010d35b393 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0897_2010050100_123_979_12196541.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0897_2010050100_123_979_12196541.zip
ic_ancil_file: ic_ancil_12196541
ifsdata_file: ifsdata_12196541
climate_data_file: clim_data_12196541
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f91813dabb16fb0399d70c6f4cfbc262 to: /var/lib/boinc/slots/2/ic_ancil_12196541.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12196541.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c93ca91bd8250914f41e47667cb5a6c to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12196541.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12196541.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12196541.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12196541.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12196541.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12196541.xml
-- Model is restarting --
Adjusting last_iter, 854, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12196541.xml
last_cpu_time: 29897
upload_file_number: 35
last_iter: 839
last_upload: 3024000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804
The child process has been launched with process id: 22093
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:22093:22093] [20230127:195058:1674874258:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  19:52:01 STEP  840 H= 840:00 +CPU= 47.490
  19:52:30 STEP  841 H= 841:00 +CPU= 26.388
  19:53:01 STEP  842 H= 842:00 +CPU= 27.221
  19:53:47 STEP  843 H= 843:00 +CPU= 41.586
  19:54:17 STEP  844 H= 844:00 +CPU= 27.232
  19:54:46 STEP  845 H= 845:00 +CPU= 26.584
  19:55:34 STEP  846 H= 846:00 +CPU= 42.646
  19:56:04 STEP  847 H= 847:00 +CPU= 26.668
  19:56:34 STEP  848 H= 848:00 +CPU= 27.239
  19:57:20 STEP  849 H= 849:00 +CPU= 41.935
  19:57:51 STEP  850 H= 850:00 +CPU= 27.600
  19:58:20 STEP  851 H= 851:00 +CPU= 26.542
  19:59:12 STEP  852 H= 852:00 +CPU= 46.739
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000852
  19:59:42 STEP  853 H= 853:00 +CPU= 26.790
  20:00:13 STEP  854 H= 854:00 +CPU= 27.440
  20:01:00 STEP  855 H= 855:00 +CPU= 42.044
  20:01:30 STEP  856 H= 856:00 +CPU= 27.507
  20:02:00 STEP  857 H= 857:00 +CPU= 26.698
  20:02:47 STEP  858 H= 858:00 +CPU= 42.583
  20:03:17 STEP  859 H= 859:00 +CPU= 26.734
  20:03:48 STEP  860 H= 860:00 +CPU= 27.758
  20:04:34 STEP  861 H= 861:00 +CPU= 42.092
  20:05:05 STEP  862 H= 862:00 +CPU= 27.539
  20:05:35 STEP  863 H= 863:00 +CPU= 27.584
  20:06:30 STEP  864 H= 864:00 +CPU= 48.676
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_35.zip
Uploading the intermediate file: upload_file_35.zip
  20:06:59 STEP  865 H= 865:00 +CPU= 26.611
  20:07:27 STEP  866 H= 866:00 +CPU= 27.432
  20:08:11 STEP  867 H= 867:00 +CPU= 41.198
  20:08:39 STEP  868 H= 868:00 +CPU= 26.791
  20:09:07 STEP  869 H= 869:00 +CPU= 26.131
  20:09:51 STEP  870 H= 870:00 +CPU= 42.289
  20:10:18 STEP  871 H= 871:00 +CPU= 25.488
  20:10:46 STEP  872 H= 872:00 +CPU= 27.004
  20:11:29 STEP  873 H= 873:00 +CPU= 40.868
  20:11:58 STEP  874 H= 874:00 +CPU= 27.071
  20:12:25 STEP  875 H= 875:00 +CPU= 26.041
  20:13:13 STEP  876 H= 876:00 +CPU= 46.185
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000876
  20:13:41 STEP  877 H= 877:00 +CPU= 26.403
  20:14:10 STEP  878 H= 878:00 +CPU= 27.091
  20:14:53 STEP  879 H= 879:00 +CPU= 40.935
  20:15:21 STEP  880 H= 880:00 +CPU= 26.843
  20:15:49 STEP  881 H= 881:00 +CPU= 26.197
  20:16:32 STEP  882 H= 882:00 +CPU= 41.883
  20:16:59 STEP  883 H= 883:00 +CPU= 25.923
  20:17:28 STEP  884 H= 884:00 +CPU= 26.854
  20:18:11 STEP  885 H= 885:00 +CPU= 41.222
  20:18:40 STEP  886 H= 886:00 +CPU= 26.904
  20:19:08 STEP  887 H= 887:00 +CPU= 27.116
  20:20:00 STEP  888 H= 888:00 +CPU= 48.483
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_36.zip
Uploading the intermediate file: upload_file_36.zip
  20:20:28 STEP  889 H= 889:00 +CPU= 25.941
  20:20:57 STEP  890 H= 890:00 +CPU= 27.331
  20:21:40 STEP  891 H= 891:00 +CPU= 41.096
  20:22:09 STEP  892 H= 892:00 +CPU= 27.041
  20:22:37 STEP  893 H= 893:00 +CPU= 26.044
  20:23:21 STEP  894 H= 894:00 +CPU= 41.743
  20:23:49 STEP  895 H= 895:00 +CPU= 25.900
  20:24:19 STEP  896 H= 896:00 +CPU= 27.127
  20:25:05 STEP  897 H= 897:00 +CPU= 41.466
  20:25:36 STEP  898 H= 898:00 +CPU= 27.210
  20:26:07 STEP  899 H= 899:00 +CPU= 26.599
  20:27:03 STEP  900 H= 900:00 +CPU= 47.025
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000900
  20:27:38 STEP  901 H= 901:00 +CPU= 27.074
  20:28:20 STEP  902 H= 902:00 +CPU= 28.350
  20:29:16 STEP  903 H= 903:00 +CPU= 42.317
  20:29:58 STEP  904 H= 904:00 +CPU= 27.616
  20:30:39 STEP  905 H= 905:00 +CPU= 27.018
  20:31:29 STEP  906 H= 906:00 +CPU= 42.902
  20:31:59 STEP  907 H= 907:00 +CPU= 26.949
  20:32:39 STEP  908 H= 908:00 +CPU= 28.005
  20:33:32 STEP  909 H= 909:00 +CPU= 42.056
  20:34:03 STEP  910 H= 910:00 +CPU= 27.904
  20:34:33 STEP  911 H= 911:00 +CPU= 27.214
  20:35:24 STEP  912 H= 912:00 +CPU= 46.717
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_37.zip
Uploading the intermediate file: upload_file_37.zip
  20:35:53 STEP  913 H= 913:00 +CPU= 25.607
  20:36:24 STEP  914 H= 914:00 +CPU= 26.781
  20:37:13 STEP  915 H= 915:00 +CPU= 40.919
  20:37:44 STEP  916 H= 916:00 +CPU= 26.553
  20:38:14 STEP  917 H= 917:00 +CPU= 25.819
  20:39:04 STEP  918 H= 918:00 +CPU= 42.368
  20:39:34 STEP  919 H= 919:00 +CPU= 25.989
  20:40:06 STEP  920 H= 920:00 +CPU= 27.480
  20:41:01 STEP  921 H= 921:00 +CPU= 41.697
  20:41:34 STEP  922 H= 922:00 +CPU= 27.496
  20:42:05 STEP  923 H= 923:00 +CPU= 27.203
  20:43:00 STEP  924 H= 924:00 +CPU= 47.348
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000924
  20:43:31 STEP  925 H= 925:00 +CPU= 27.155
  20:44:05 STEP  926 H= 926:00 +CPU= 28.406
  20:44:54 STEP  927 H= 927:00 +CPU= 42.044
  20:45:27 STEP  928 H= 928:00 +CPU= 28.041
  20:46:03 STEP  929 H= 929:00 +CPU= 27.301
  20:46:55 STEP  930 H= 930:00 +CPU= 43.664
  20:47:26 STEP  931 H= 931:00 +CPU= 26.857
  20:47:58 STEP  932 H= 932:00 +CPU= 27.835
  20:48:47 STEP  933 H= 933:00 +CPU= 42.577
  20:49:19 STEP  934 H= 934:00 +CPU= 27.865
  20:49:51 STEP  935 H= 935:00 +CPU= 27.859
  20:50:54 STEP  936 H= 936:00 +CPU= 49.706
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_38.zip
Uploading the intermediate file: upload_file_38.zip
  20:51:26 STEP  937 H= 937:00 +CPU= 26.996
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0897
(argv4) batchid: 979
(argv5) wuid: 12196541
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03dbe9c1a0aea012c6b90d010d35b393 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0897_2010050100_123_979_12196541.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0897_2010050100_123_979_12196541.zip
ic_ancil_file: ic_ancil_12196541
ifsdata_file: ifsdata_12196541
climate_data_file: clim_data_12196541
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f91813dabb16fb0399d70c6f4cfbc262 to: /var/lib/boinc/slots/2/ic_ancil_12196541.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12196541.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c93ca91bd8250914f41e47667cb5a6c to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12196541.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12196541.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12196541.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12196541.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12196541.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12196541.xml
-- Model is restarting --
Adjusting last_iter, 938, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12196541.xml
last_cpu_time: 33090
upload_file_number: 39
last_iter: 935
last_upload: 3369600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804
The child process has been launched with process id: 16782
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:16782:16782] [20230127:214126:1674880886:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:42:20 STEP  936 H= 936:00 +CPU= 47.427
  21:42:49 STEP  937 H= 937:00 +CPU= 27.163
  21:43:18 STEP  938 H= 938:00 +CPU= 27.836
  21:44:03 STEP  939 H= 939:00 +CPU= 42.524
  21:44:32 STEP  940 H= 940:00 +CPU= 27.744
  21:45:01 STEP  941 H= 941:00 +CPU= 27.258
  21:45:47 STEP  942 H= 942:00 +CPU= 43.592
  21:46:15 STEP  943 H= 943:00 +CPU= 26.836
  21:46:45 STEP  944 H= 944:00 +CPU= 27.777
  21:47:30 STEP  945 H= 945:00 +CPU= 42.388
  21:48:00 STEP  946 H= 946:00 +CPU= 27.714
  21:48:28 STEP  947 H= 947:00 +CPU= 26.884
  21:49:19 STEP  948 H= 948:00 +CPU= 47.040
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000948
  21:49:47 STEP  949 H= 949:00 +CPU= 27.085
  21:50:17 STEP  950 H= 950:00 +CPU= 28.014
  21:51:02 STEP  951 H= 951:00 +CPU= 42.236
  21:51:31 STEP  952 H= 952:00 +CPU= 27.745
  21:52:00 STEP  953 H= 953:00 +CPU= 26.884
  21:52:48 STEP  954 H= 954:00 +CPU= 43.406
  21:53:17 STEP  955 H= 955:00 +CPU= 26.982
  21:53:46 STEP  956 H= 956:00 +CPU= 27.815
  21:54:32 STEP  957 H= 957:00 +CPU= 42.053
  21:55:01 STEP  958 H= 958:00 +CPU= 27.859
  21:55:30 STEP  959 H= 959:00 +CPU= 27.353
  21:56:22 STEP  960 H= 960:00 +CPU= 48.774
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_39.zip
Uploading the intermediate file: upload_file_39.zip
  21:56:49 STEP  961 H= 961:00 +CPU= 25.622
  21:57:17 STEP  962 H= 962:00 +CPU= 26.811
  21:58:02 STEP  963 H= 963:00 +CPU= 42.841
  21:58:31 STEP  964 H= 964:00 +CPU= 27.624
  21:59:00 STEP  965 H= 965:00 +CPU= 27.433
  21:59:46 STEP  966 H= 966:00 +CPU= 43.768
  22:00:15 STEP  967 H= 967:00 +CPU= 27.632
  22:00:44 STEP  968 H= 968:00 +CPU= 27.580
  22:01:31 STEP  969 H= 969:00 +CPU= 42.746
  22:02:01 STEP  970 H= 970:00 +CPU= 28.434
  22:02:30 STEP  971 H= 971:00 +CPU= 27.571
  22:03:20 STEP  972 H= 972:00 +CPU= 47.464
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000972
  22:03:50 STEP  973 H= 973:00 +CPU= 27.594
  22:04:19 STEP  974 H= 974:00 +CPU= 28.003
  22:05:04 STEP  975 H= 975:00 +CPU= 42.694
  22:05:33 STEP  976 H= 976:00 +CPU= 27.229
  22:06:01 STEP  977 H= 977:00 +CPU= 26.764
  22:06:48 STEP  978 H= 978:00 +CPU= 43.610
  22:07:17 STEP  979 H= 979:00 +CPU= 27.078
  22:07:47 STEP  980 H= 980:00 +CPU= 28.072
  22:08:32 STEP  981 H= 981:00 +CPU= 42.500
  22:09:02 STEP  982 H= 982:00 +CPU= 28.300
  22:09:32 STEP  983 H= 983:00 +CPU= 28.145
  22:10:25 STEP  984 H= 984:00 +CPU= 49.288
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_40.zip
Uploading the intermediate file: upload_file_40.zip
  22:10:54 STEP  985 H= 985:00 +CPU= 26.823
  22:11:24 STEP  986 H= 986:00 +CPU= 28.065
  22:12:09 STEP  987 H= 987:00 +CPU= 42.552
  22:12:39 STEP  988 H= 988:00 +CPU= 28.391
  22:13:08 STEP  989 H= 989:00 +CPU= 26.805
  22:13:54 STEP  990 H= 990:00 +CPU= 43.409
  22:14:23 STEP  991 H= 991:00 +CPU= 26.775
  22:14:53 STEP  992 H= 992:00 +CPU= 27.814
  22:15:38 STEP  993 H= 993:00 +CPU= 42.283
  22:16:08 STEP  994 H= 994:00 +CPU= 28.250
  22:16:36 STEP  995 H= 995:00 +CPU= 26.631
  22:17:25 STEP  996 H= 996:00 +CPU= 45.843
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+000996
  22:17:53 STEP  997 H= 997:00 +CPU= 26.236
  22:18:21 STEP  998 H= 998:00 +CPU= 26.693
  22:19:06 STEP  999 H= 999:00 +CPU= 41.083
  22:19:35 STEP 1000 H=1000:00 +CPU= 27.015
  22:20:03 STEP 1001 H=1001:00 +CPU= 26.136
  22:20:47 STEP 1002 H=1002:00 +CPU= 41.623
  22:21:14 STEP 1003 H=1003:00 +CPU= 25.580
  22:21:42 STEP 1004 H=1004:00 +CPU= 26.298
  22:22:25 STEP 1005 H=1005:00 +CPU= 40.882
  22:22:53 STEP 1006 H=1006:00 +CPU= 26.336
  22:23:21 STEP 1007 H=1007:00 +CPU= 26.318
  22:24:11 STEP 1008 H=1008:00 +CPU= 47.161
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_41.zip
Uploading the intermediate file: upload_file_41.zip
  22:24:37 STEP 1009 H=1009:00 +CPU= 25.397
  22:25:05 STEP 1010 H=1010:00 +CPU= 26.300
  22:25:48 STEP 1011 H=1011:00 +CPU= 40.600
  22:26:15 STEP 1012 H=1012:00 +CPU= 26.518
  22:26:42 STEP 1013 H=1013:00 +CPU= 25.270
  22:27:26 STEP 1014 H=1014:00 +CPU= 41.353
  22:27:53 STEP 1015 H=1015:00 +CPU= 25.602
  22:28:21 STEP 1016 H=1016:00 +CPU= 26.417
  22:29:04 STEP 1017 H=1017:00 +CPU= 40.656
  22:29:32 STEP 1018 H=1018:00 +CPU= 26.434
  22:29:59 STEP 1019 H=1019:00 +CPU= 25.788
  22:30:46 STEP 1020 H=1020:00 +CPU= 44.970
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001020
  22:31:13 STEP 1021 H=1021:00 +CPU= 25.647
  22:31:42 STEP 1022 H=1022:00 +CPU= 26.663
  22:32:24 STEP 1023 H=1023:00 +CPU= 40.388
  22:32:52 STEP 1024 H=1024:00 +CPU= 26.447
  22:33:19 STEP 1025 H=1025:00 +CPU= 25.413
  22:34:03 STEP 1026 H=1026:00 +CPU= 41.698
  22:34:30 STEP 1027 H=1027:00 +CPU= 25.714
  22:34:57 STEP 1028 H=1028:00 +CPU= 26.136
  22:35:40 STEP 1029 H=1029:00 +CPU= 40.498
  22:36:08 STEP 1030 H=1030:00 +CPU= 26.274
  22:36:35 STEP 1031 H=1031:00 +CPU= 26.239
  22:37:25 STEP 1032 H=1032:00 +CPU= 47.028
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804_42.zip
Uploading the intermediate file: upload_file_42.zip
  22:37:52 STEP 1033 H=1033:00 +CPU= 25.366
  22:38:20 STEP 1034 H=1034:00 +CPU= 26.577
  22:39:03 STEP 1035 H=1035:00 +CPU= 40.262
  22:39:31 STEP 1036 H=1036:00 +CPU= 26.819
  22:39:58 STEP 1037 H=1037:00 +CPU= 25.760
  22:40:42 STEP 1038 H=1038:00 +CPU= 41.552
  22:41:09 STEP 1039 H=1039:00 +CPU= 25.437
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0897
(argv4) batchid: 979
(argv5) wuid: 12196541
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/2
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12196541
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_03dbe9c1a0aea012c6b90d010d35b393 to: /var/lib/boinc/slots/2/oifs_43r3_ps_0897_2010050100_123_979_12196541.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/2/oifs_43r3_ps_0897_2010050100_123_979_12196541.zip
ic_ancil_file: ic_ancil_12196541
ifsdata_file: ifsdata_12196541
climate_data_file: clim_data_12196541
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f91813dabb16fb0399d70c6f4cfbc262 to: /var/lib/boinc/slots/2/ic_ancil_12196541.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/2/ic_ancil_12196541.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c93ca91bd8250914f41e47667cb5a6c to: /var/lib/boinc/slots/2/ifsdata/ifsdata_12196541.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/2/ifsdata/ifsdata_12196541.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/2/159l_2/clim_data_12196541.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/2/159l_2/clim_data_12196541.zip
Checking for progress XML file: /var/lib/boinc/slots/2/progress_file_12196541.xml
Opened progress file ok : /var/lib/boinc/slots/2/progress_file_12196541.xml
-- Model is restarting --
Adjusting last_iter, 1040, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/2/progress_file_12196541.xml
last_cpu_time: 36481
upload_file_number: 43
last_iter: 1031
last_upload: 3715200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0897_2010050100_123_979_12196541_1_r155370804
The child process has been launched with process id: 27157
Executing the command: /var/lib/boinc/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:gokuraku:1:1:27157:27157] [20230128:004112:1674891672:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:42:02 STEP 1032 H=1032:00 +CPU= 45.785
  00:42:29 STEP 1033 H=1033:00 +CPU= 25.812
  00:42:57 STEP 1034 H=1034:00 +CPU= 26.662
  00:43:40 STEP 1035 H=1035:00 +CPU= 40.459
  00:44:08 STEP 1036 H=1036:00 +CPU= 26.672
  00:44:35 STEP 1037 H=1037:00 +CPU= 25.701
  00:45:19 STEP 1038 H=1038:00 +CPU= 41.580
  00:45:46 STEP 1039 H=1039:00 +CPU= 26.056
  00:46:14 STEP 1040 H=1040:00 +CPU= 26.801
  00:46:57 STEP 1041 H=1041:00 +CPU= 40.647
  00:47:25 STEP 1042 H=1042:00 +CPU= 26.344
  00:47:52 STEP 1043 H=1043:00 +CPU= 25.566
  00:48:40 STEP 1044 H=1044:00 +CPU= 45.684
Moving to projects directory: /var/lib/boinc/slots/2/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/2/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/2/ICMUAhq0f+001044
  00:49:07 STEP 1045 H=1045:00 +CPU= 25.796
  00:49:35 STEP 1046 H=1046:00 +CPU= 26.640
  00:50:19 STEP 1047 H=1047:00 +CPU= 41.057
  00:50:47 STEP 1048 H=1048:00 +CPU= 26.820
  00:51:14 STEP 1049 H=1049:00 +CPU= 26.020
  00:51:58 STEP 1050 H=1050:00 +CPU= 41.552
  00:52:25 STEP 1051 H=1051:00 +CPU= 25.821
  00:52:53 STEP 1052 H=1052:00 +CPU= 26.681
  00:53:36 STEP 1053 H=1053:00 +CPU= 40.521
  00:54:05 STEP 1054 H=1054:00 +CPU= 26.864
  00:54:33 STEP 1055 H=1055:00 +CPU= 26.508
  00:55:23 STEP 1056 H=1056:00 +CPU= 47.948
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:42:25 1493665 22303871 oifs_43r3_ps_0897_2010050100_123_979_12196541_1 1,641,600 15,652 0.0095


©2024 cpdn.org