climateprediction.net home page
Task 22302631

Task 22302631

Name oifs_43r3_ps_0720_2019050100_123_988_12205364_1
Workunit 12205364
Created 24 Jan 2023, 0:30:19 UTC
Sent 24 Jan 2023, 0:37:57 UTC
Report deadline 25 Mar 2023, 0:37:57 UTC
Received 24 Jan 2023, 23:51:32 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1537854
Run time 22 hours 14 min 11 sec
CPU time 21 hours 10 min 59 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 3.60 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,393.08 MB
Peak swap size 4,974.27 MB
Peak disk usage 2,765.70 MB
Stderr 
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
hq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000744
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000756
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000756
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000756
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_31.zip
Uploading the intermediate file: upload_file_31.zip
  14:30:31 STEP  769 H= 769:00 +CPU= 42.965
  14:31:18 STEP  770 H= 770:00 +CPU= 44.361
  14:32:28 STEP  771 H= 771:00 +CPU= 67.253
  14:33:15 STEP  772 H= 772:00 +CPU= 44.454
  14:34:00 STEP  773 H= 773:00 +CPU= 42.964
  14:35:13 STEP  774 H= 774:00 +CPU= 69.433
  14:35:58 STEP  775 H= 775:00 +CPU= 43.135
  14:36:45 STEP  776 H= 776:00 +CPU= 44.506
  14:37:57 STEP  777 H= 777:00 +CPU= 68.050
  14:38:44 STEP  778 H= 778:00 +CPU= 44.415
  14:39:29 STEP  779 H= 779:00 +CPU= 43.334
  14:40:49 STEP  780 H= 780:00 +CPU= 75.583
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000780
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000780
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000780
  14:41:34 STEP  781 H= 781:00 +CPU= 43.186
  14:42:21 STEP  782 H= 782:00 +CPU= 44.489
  14:43:32 STEP  783 H= 783:00 +CPU= 67.849
  14:44:19 STEP  784 H= 784:00 +CPU= 44.577
  14:45:04 STEP  785 H= 785:00 +CPU= 43.275
  14:46:17 STEP  786 H= 786:00 +CPU= 69.048
  14:47:02 STEP  787 H= 787:00 +CPU= 43.159
  14:47:48 STEP  788 H= 788:00 +CPU= 44.420
  14:48:59 STEP  789 H= 789:00 +CPU= 67.204
  14:49:45 STEP  790 H= 790:00 +CPU= 44.319
  14:50:32 STEP  791 H= 791:00 +CPU= 44.371
  14:51:57 STEP  792 H= 792:00 +CPU= 80.052
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000792
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000792
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000768
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000780
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000780
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000780
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_32.zip
Uploading the intermediate file: upload_file_32.zip
  14:52:42 STEP  793 H= 793:00 +CPU= 43.055
  14:53:28 STEP  794 H= 794:00 +CPU= 44.491
  14:54:40 STEP  795 H= 795:00 +CPU= 67.683
  14:55:26 STEP  796 H= 796:00 +CPU= 44.491
  14:56:12 STEP  797 H= 797:00 +CPU= 42.947
  14:57:25 STEP  798 H= 798:00 +CPU= 69.521
  14:58:10 STEP  799 H= 799:00 +CPU= 42.970
  14:58:57 STEP  800 H= 800:00 +CPU= 44.622
  15:00:09 STEP  801 H= 801:00 +CPU= 67.537
  15:00:55 STEP  802 H= 802:00 +CPU= 44.498
  15:01:40 STEP  803 H= 803:00 +CPU= 43.052
  15:02:59 STEP  804 H= 804:00 +CPU= 75.704
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000804
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000804
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000804
  15:03:45 STEP  805 H= 805:00 +CPU= 43.398
  15:04:32 STEP  806 H= 806:00 +CPU= 44.579
  15:05:43 STEP  807 H= 807:00 +CPU= 68.056
  15:06:30 STEP  808 H= 808:00 +CPU= 44.236
  15:07:15 STEP  809 H= 809:00 +CPU= 43.235
  15:08:27 STEP  810 H= 810:00 +CPU= 69.229
  15:09:12 STEP  811 H= 811:00 +CPU= 43.114
  15:09:59 STEP  812 H= 812:00 +CPU= 44.307
  15:11:10 STEP  813 H= 813:00 +CPU= 67.954
  15:11:56 STEP  814 H= 814:00 +CPU= 44.323
  15:12:43 STEP  815 H= 815:00 +CPU= 44.540
  15:14:08 STEP  816 H= 816:00 +CPU= 79.793
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000816
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000816
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000792
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000804
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000804
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000804
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_33.zip
Uploading the intermediate file: upload_file_33.zip
  15:14:53 STEP  817 H= 817:00 +CPU= 43.188
  15:15:40 STEP  818 H= 818:00 +CPU= 44.462
  15:16:51 STEP  819 H= 819:00 +CPU= 67.397
  15:17:38 STEP  820 H= 820:00 +CPU= 44.451
  15:18:24 STEP  821 H= 821:00 +CPU= 42.708
  15:19:37 STEP  822 H= 822:00 +CPU= 69.593
  15:20:22 STEP  823 H= 823:00 +CPU= 43.007
  15:21:09 STEP  824 H= 824:00 +CPU= 44.479
  15:22:20 STEP  825 H= 825:00 +CPU= 68.182
  15:23:07 STEP  826 H= 826:00 +CPU= 44.650
  15:23:52 STEP  827 H= 827:00 +CPU= 43.393
  15:25:12 STEP  828 H= 828:00 +CPU= 75.941
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000828
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000828
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000828
  15:25:57 STEP  829 H= 829:00 +CPU= 43.245
  15:26:44 STEP  830 H= 830:00 +CPU= 44.412
  15:27:55 STEP  831 H= 831:00 +CPU= 68.040
  15:28:41 STEP  832 H= 832:00 +CPU= 44.623
  15:29:27 STEP  833 H= 833:00 +CPU= 43.222
  15:30:39 STEP  834 H= 834:00 +CPU= 69.243
  15:31:24 STEP  835 H= 835:00 +CPU= 43.266
  15:32:11 STEP  836 H= 836:00 +CPU= 44.455
  15:33:22 STEP  837 H= 837:00 +CPU= 68.096
  15:34:09 STEP  838 H= 838:00 +CPU= 44.166
  15:34:56 STEP  839 H= 839:00 +CPU= 44.665
  15:36:22 STEP  840 H= 840:00 +CPU= 79.587
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000840
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000840
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000816
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000828
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000828
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000828
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_34.zip
Uploading the intermediate file: upload_file_34.zip
  15:37:07 STEP  841 H= 841:00 +CPU= 43.246
  15:37:54 STEP  842 H= 842:00 +CPU= 44.585
  15:39:05 STEP  843 H= 843:00 +CPU= 67.925
  15:39:52 STEP  844 H= 844:00 +CPU= 44.452
  15:40:38 STEP  845 H= 845:00 +CPU= 42.882
  15:41:50 STEP  846 H= 846:00 +CPU= 69.428
  15:42:36 STEP  847 H= 847:00 +CPU= 43.317
  15:43:22 STEP  848 H= 848:00 +CPU= 44.503
  15:44:33 STEP  849 H= 849:00 +CPU= 68.087
  15:45:20 STEP  850 H= 850:00 +CPU= 44.575
  15:46:05 STEP  851 H= 851:00 +CPU= 43.332
  15:47:25 STEP  852 H= 852:00 +CPU= 76.017
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000852
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000852
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000852
  15:48:10 STEP  853 H= 853:00 +CPU= 43.409
  15:48:57 STEP  854 H= 854:00 +CPU= 44.494
  15:50:08 STEP  855 H= 855:00 +CPU= 67.652
  15:50:55 STEP  856 H= 856:00 +CPU= 44.478
  15:51:40 STEP  857 H= 857:00 +CPU= 43.289
  15:52:53 STEP  858 H= 858:00 +CPU= 69.174
  15:53:38 STEP  859 H= 859:00 +CPU= 43.356
  15:54:26 STEP  860 H= 860:00 +CPU= 44.291
  15:55:37 STEP  861 H= 861:00 +CPU= 67.985
  15:56:24 STEP  862 H= 862:00 +CPU= 44.308
  15:57:11 STEP  863 H= 863:00 +CPU= 44.603
  15:58:36 STEP  864 H= 864:00 +CPU= 79.533
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000864
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000864
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000840
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000852
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000852
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000852
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_35.zip
Uploading the intermediate file: upload_file_35.zip
  15:59:21 STEP  865 H= 865:00 +CPU= 43.209
  16:00:08 STEP  866 H= 866:00 +CPU= 44.111
  16:01:19 STEP  867 H= 867:00 +CPU= 68.056
  16:02:06 STEP  868 H= 868:00 +CPU= 44.498
  16:02:51 STEP  869 H= 869:00 +CPU= 43.263
  16:04:04 STEP  870 H= 870:00 +CPU= 69.632
  16:04:49 STEP  871 H= 871:00 +CPU= 43.163
  16:05:36 STEP  872 H= 872:00 +CPU= 44.539
  16:06:47 STEP  873 H= 873:00 +CPU= 67.949
  16:07:34 STEP  874 H= 874:00 +CPU= 44.672
  16:08:20 STEP  875 H= 875:00 +CPU= 43.138
  16:09:39 STEP  876 H= 876:00 +CPU= 75.946
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000876
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000876
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000876
  16:10:25 STEP  877 H= 877:00 +CPU= 43.310
  16:11:11 STEP  878 H= 878:00 +CPU= 44.562
  16:12:23 STEP  879 H= 879:00 +CPU= 68.061
  16:13:09 STEP  880 H= 880:00 +CPU= 44.521
  16:13:54 STEP  881 H= 881:00 +CPU= 43.166
  16:15:07 STEP  882 H= 882:00 +CPU= 69.213
  16:15:53 STEP  883 H= 883:00 +CPU= 43.297
  16:16:40 STEP  884 H= 884:00 +CPU= 44.036
  16:17:51 STEP  885 H= 885:00 +CPU= 68.076
  16:18:38 STEP  886 H= 886:00 +CPU= 44.428
  16:19:25 STEP  887 H= 887:00 +CPU= 44.668
  16:20:50 STEP  888 H= 888:00 +CPU= 79.551
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000888
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000888
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000864
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000876
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000876
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000876
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_36.zip
Uploading the intermediate file: upload_file_36.zip
  16:21:35 STEP  889 H= 889:00 +CPU= 43.282
  16:22:22 STEP  890 H= 890:00 +CPU= 44.356
  16:23:33 STEP  891 H= 891:00 +CPU= 68.054
  16:24:20 STEP  892 H= 892:00 +CPU= 44.597
  16:25:06 STEP  893 H= 893:00 +CPU= 43.325
  16:26:19 STEP  894 H= 894:00 +CPU= 69.383
  16:27:04 STEP  895 H= 895:00 +CPU= 43.245
  16:27:51 STEP  896 H= 896:00 +CPU= 44.559
  16:29:02 STEP  897 H= 897:00 +CPU= 68.043
  16:29:49 STEP  898 H= 898:00 +CPU= 44.618
  16:30:34 STEP  899 H= 899:00 +CPU= 43.126
  16:31:54 STEP  900 H= 900:00 +CPU= 76.102
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000900
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000900
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000900
  16:32:39 STEP  901 H= 901:00 +CPU= 43.073
  16:33:26 STEP  902 H= 902:00 +CPU= 44.646
  16:34:38 STEP  903 H= 903:00 +CPU= 67.867
  16:35:24 STEP  904 H= 904:00 +CPU= 44.535
  16:36:10 STEP  905 H= 905:00 +CPU= 42.762
  16:37:23 STEP  906 H= 906:00 +CPU= 69.274
  16:38:08 STEP  907 H= 907:00 +CPU= 43.021
  16:38:55 STEP  908 H= 908:00 +CPU= 44.470
  16:40:06 STEP  909 H= 909:00 +CPU= 67.686
  16:40:52 STEP  910 H= 910:00 +CPU= 44.447
  16:41:39 STEP  911 H= 911:00 +CPU= 44.629
  16:43:05 STEP  912 H= 912:00 +CPU= 79.999
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000912
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000912
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000888
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000900
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000900
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000900
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_37.zip
Uploading the intermediate file: upload_file_37.zip
  16:43:50 STEP  913 H= 913:00 +CPU= 43.237
  16:44:37 STEP  914 H= 914:00 +CPU= 44.605
  16:45:48 STEP  915 H= 915:00 +CPU= 67.911
  16:46:34 STEP  916 H= 916:00 +CPU= 44.373
  16:47:19 STEP  917 H= 917:00 +CPU= 43.231
  16:48:32 STEP  918 H= 918:00 +CPU= 69.331
  16:49:17 STEP  919 H= 919:00 +CPU= 43.282
  16:50:04 STEP  920 H= 920:00 +CPU= 44.235
  16:51:15 STEP  921 H= 921:00 +CPU= 68.049
  16:52:02 STEP  922 H= 922:00 +CPU= 44.623
  16:52:47 STEP  923 H= 923:00 +CPU= 43.332
  16:54:08 STEP  924 H= 924:00 +CPU= 75.982
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000924
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000924
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000924
  16:54:53 STEP  925 H= 925:00 +CPU= 43.467
  16:55:40 STEP  926 H= 926:00 +CPU= 44.643
  16:56:52 STEP  927 H= 927:00 +CPU= 67.613
  16:57:38 STEP  928 H= 928:00 +CPU= 44.531
  16:58:24 STEP  929 H= 929:00 +CPU= 43.027
  16:59:37 STEP  930 H= 930:00 +CPU= 69.410
  17:00:23 STEP  931 H= 931:00 +CPU= 42.852
  17:01:09 STEP  932 H= 932:00 +CPU= 44.533
  17:02:21 STEP  933 H= 933:00 +CPU= 68.077
  17:03:07 STEP  934 H= 934:00 +CPU= 44.311
  17:03:54 STEP  935 H= 935:00 +CPU= 44.600
  17:05:19 STEP  936 H= 936:00 +CPU= 79.934
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000936
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000936
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000912
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000924
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000924
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000924
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_38.zip
Uploading the intermediate file: upload_file_38.zip
  17:06:05 STEP  937 H= 937:00 +CPU= 43.175
  17:06:51 STEP  938 H= 938:00 +CPU= 44.390
  17:08:02 STEP  939 H= 939:00 +CPU= 67.690
  17:08:48 STEP  940 H= 940:00 +CPU= 44.521
  17:09:34 STEP  941 H= 941:00 +CPU= 43.186
  17:10:47 STEP  942 H= 942:00 +CPU= 69.417
  17:11:32 STEP  943 H= 943:00 +CPU= 43.266
  17:12:19 STEP  944 H= 944:00 +CPU= 44.635
  17:13:30 STEP  945 H= 945:00 +CPU= 68.099
  17:14:17 STEP  946 H= 946:00 +CPU= 44.417
  17:15:02 STEP  947 H= 947:00 +CPU= 43.427
  17:16:22 STEP  948 H= 948:00 +CPU= 75.598
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000948
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000948
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000948
  17:17:08 STEP  949 H= 949:00 +CPU= 43.244
  17:17:54 STEP  950 H= 950:00 +CPU= 44.618
  17:19:06 STEP  951 H= 951:00 +CPU= 68.009
  17:19:52 STEP  952 H= 952:00 +CPU= 44.546
  17:20:38 STEP  953 H= 953:00 +CPU= 42.936
  17:21:51 STEP  954 H= 954:00 +CPU= 69.383
  17:22:36 STEP  955 H= 955:00 +CPU= 43.200
  17:23:22 STEP  956 H= 956:00 +CPU= 44.472
  17:24:33 STEP  957 H= 957:00 +CPU= 67.776
  17:25:20 STEP  958 H= 958:00 +CPU= 44.406
  17:26:07 STEP  959 H= 959:00 +CPU= 44.572
  17:27:31 STEP  960 H= 960:00 +CPU= 80.004
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000960
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000960
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000936
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000948
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000948
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000948
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_39.zip
Uploading the intermediate file: upload_file_39.zip
Uploading trickle at timestep: 3456000
  17:28:16 STEP  961 H= 961:00 +CPU= 43.119
  17:29:03 STEP  962 H= 962:00 +CPU= 44.413
  17:30:14 STEP  963 H= 963:00 +CPU= 67.847
  17:31:01 STEP  964 H= 964:00 +CPU= 44.444
  17:31:46 STEP  965 H= 965:00 +CPU= 43.276
  17:32:59 STEP  966 H= 966:00 +CPU= 69.437
  17:33:44 STEP  967 H= 967:00 +CPU= 43.241
  17:34:31 STEP  968 H= 968:00 +CPU= 44.464
  17:35:42 STEP  969 H= 969:00 +CPU= 67.953
  17:36:29 STEP  970 H= 970:00 +CPU= 44.384
  17:37:14 STEP  971 H= 971:00 +CPU= 43.242
  17:38:34 STEP  972 H= 972:00 +CPU= 76.076
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000972
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000972
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000972
  17:39:19 STEP  973 H= 973:00 +CPU= 43.362
  17:40:06 STEP  974 H= 974:00 +CPU= 44.552
  17:41:17 STEP  975 H= 975:00 +CPU= 68.048
  17:42:04 STEP  976 H= 976:00 +CPU= 44.585
  17:42:49 STEP  977 H= 977:00 +CPU= 43.395
  17:44:02 STEP  978 H= 978:00 +CPU= 69.401
  17:44:47 STEP  979 H= 979:00 +CPU= 43.205
  17:45:33 STEP  980 H= 980:00 +CPU= 44.484
  17:46:44 STEP  981 H= 981:00 +CPU= 67.874
  17:47:31 STEP  982 H= 982:00 +CPU= 44.457
  17:48:18 STEP  983 H= 983:00 +CPU= 44.544
  17:49:42 STEP  984 H= 984:00 +CPU= 79.882
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000984
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000984
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000960
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000972
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000972
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000972
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_40.zip
Uploading the intermediate file: upload_file_40.zip
  17:50:28 STEP  985 H= 985:00 +CPU= 43.173
  17:51:14 STEP  986 H= 986:00 +CPU= 44.469
  17:52:25 STEP  987 H= 987:00 +CPU= 67.999
  17:53:12 STEP  988 H= 988:00 +CPU= 44.525
  17:53:57 STEP  989 H= 989:00 +CPU= 43.174
  17:55:10 STEP  990 H= 990:00 +CPU= 69.404
  17:55:55 STEP  991 H= 991:00 +CPU= 43.331
  17:56:42 STEP  992 H= 992:00 +CPU= 44.458
  17:57:53 STEP  993 H= 993:00 +CPU= 68.005
  17:58:40 STEP  994 H= 994:00 +CPU= 44.518
  17:59:26 STEP  995 H= 995:00 +CPU= 43.239
  18:00:45 STEP  996 H= 996:00 +CPU= 75.851
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+000996
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+000996
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+000996
  18:01:30 STEP  997 H= 997:00 +CPU= 43.328
  18:02:17 STEP  998 H= 998:00 +CPU= 44.458
  18:03:28 STEP  999 H= 999:00 +CPU= 68.068
  18:04:15 STEP 1000 H=1000:00 +CPU= 44.491
  18:05:00 STEP 1001 H=1001:00 +CPU= 43.207
  18:06:13 STEP 1002 H=1002:00 +CPU= 69.429
  18:06:58 STEP 1003 H=1003:00 +CPU= 43.268
  18:07:45 STEP 1004 H=1004:00 +CPU= 44.426
  18:08:56 STEP 1005 H=1005:00 +CPU= 67.947
  18:09:42 STEP 1006 H=1006:00 +CPU= 44.353
  18:10:29 STEP 1007 H=1007:00 +CPU= 44.612
  18:11:53 STEP 1008 H=1008:00 +CPU= 79.693
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001008
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001008
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000984
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+000996
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+000996
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+000996
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_41.zip
Uploading the intermediate file: upload_file_41.zip
  18:12:39 STEP 1009 H=1009:00 +CPU= 43.018
  18:13:25 STEP 1010 H=1010:00 +CPU= 44.400
  18:14:36 STEP 1011 H=1011:00 +CPU= 67.724
  18:15:22 STEP 1012 H=1012:00 +CPU= 44.452
  18:16:08 STEP 1013 H=1013:00 +CPU= 43.090
  18:17:20 STEP 1014 H=1014:00 +CPU= 69.203
  18:18:05 STEP 1015 H=1015:00 +CPU= 43.321
  18:18:52 STEP 1016 H=1016:00 +CPU= 44.377
  18:20:03 STEP 1017 H=1017:00 +CPU= 67.908
  18:20:49 STEP 1018 H=1018:00 +CPU= 44.423
  18:21:35 STEP 1019 H=1019:00 +CPU= 43.238
  18:22:54 STEP 1020 H=1020:00 +CPU= 75.765
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001020
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001020
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001020
  18:23:39 STEP 1021 H=1021:00 +CPU= 43.255
  18:24:26 STEP 1022 H=1022:00 +CPU= 44.416
  18:25:37 STEP 1023 H=1023:00 +CPU= 67.937
  18:26:23 STEP 1024 H=1024:00 +CPU= 44.563
  18:27:09 STEP 1025 H=1025:00 +CPU= 43.102
  18:28:21 STEP 1026 H=1026:00 +CPU= 69.293
  18:29:06 STEP 1027 H=1027:00 +CPU= 43.221
  18:29:53 STEP 1028 H=1028:00 +CPU= 44.459
  18:31:04 STEP 1029 H=1029:00 +CPU= 68.020
  18:31:51 STEP 1030 H=1030:00 +CPU= 44.425
  18:32:37 STEP 1031 H=1031:00 +CPU= 44.582
  18:34:02 STEP 1032 H=1032:00 +CPU= 79.640
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001032
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001032
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001008
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001020
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001020
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001020
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_42.zip
Uploading the intermediate file: upload_file_42.zip
  18:34:47 STEP 1033 H=1033:00 +CPU= 43.094
  18:35:34 STEP 1034 H=1034:00 +CPU= 44.450
  18:36:45 STEP 1035 H=1035:00 +CPU= 67.783
  18:37:31 STEP 1036 H=1036:00 +CPU= 44.513
  18:38:16 STEP 1037 H=1037:00 +CPU= 43.135
  18:39:29 STEP 1038 H=1038:00 +CPU= 69.085
  18:40:14 STEP 1039 H=1039:00 +CPU= 43.074
  18:41:00 STEP 1040 H=1040:00 +CPU= 44.522
  18:42:11 STEP 1041 H=1041:00 +CPU= 67.886
  18:42:58 STEP 1042 H=1042:00 +CPU= 44.404
  18:43:43 STEP 1043 H=1043:00 +CPU= 43.305
  18:45:03 STEP 1044 H=1044:00 +CPU= 76.051
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001044
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001044
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001044
  18:45:48 STEP 1045 H=1045:00 +CPU= 43.226
  18:46:35 STEP 1046 H=1046:00 +CPU= 44.572
  18:47:46 STEP 1047 H=1047:00 +CPU= 67.767
  18:48:33 STEP 1048 H=1048:00 +CPU= 44.445
  18:49:18 STEP 1049 H=1049:00 +CPU= 43.188
  18:50:30 STEP 1050 H=1050:00 +CPU= 69.287
  18:51:16 STEP 1051 H=1051:00 +CPU= 43.229
  18:52:02 STEP 1052 H=1052:00 +CPU= 44.498
  18:53:13 STEP 1053 H=1053:00 +CPU= 67.816
  18:54:00 STEP 1054 H=1054:00 +CPU= 44.405
  18:54:46 STEP 1055 H=1055:00 +CPU= 44.457
  18:56:11 STEP 1056 H=1056:00 +CPU= 79.950
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001056
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001056
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001032
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001044
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001044
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001044
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_43.zip
Uploading the intermediate file: upload_file_43.zip
  18:56:56 STEP 1057 H=1057:00 +CPU= 43.077
  18:57:43 STEP 1058 H=1058:00 +CPU= 44.453
  18:58:54 STEP 1059 H=1059:00 +CPU= 68.026
  18:59:41 STEP 1060 H=1060:00 +CPU= 44.383
  19:00:26 STEP 1061 H=1061:00 +CPU= 43.266
  19:01:39 STEP 1062 H=1062:00 +CPU= 69.318
  19:02:24 STEP 1063 H=1063:00 +CPU= 43.212
  19:03:11 STEP 1064 H=1064:00 +CPU= 44.496
  19:04:22 STEP 1065 H=1065:00 +CPU= 67.971
  19:05:09 STEP 1066 H=1066:00 +CPU= 44.510
  19:05:54 STEP 1067 H=1067:00 +CPU= 43.338
  19:07:14 STEP 1068 H=1068:00 +CPU= 75.835
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001068
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001068
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001068
  19:07:59 STEP 1069 H=1069:00 +CPU= 43.254
  19:08:46 STEP 1070 H=1070:00 +CPU= 44.574
  19:09:57 STEP 1071 H=1071:00 +CPU= 67.784
  19:10:43 STEP 1072 H=1072:00 +CPU= 44.335
  19:11:28 STEP 1073 H=1073:00 +CPU= 43.209
  19:12:41 STEP 1074 H=1074:00 +CPU= 69.203
  19:13:26 STEP 1075 H=1075:00 +CPU= 43.351
  19:14:13 STEP 1076 H=1076:00 +CPU= 44.321
  19:15:24 STEP 1077 H=1077:00 +CPU= 67.746
  19:16:10 STEP 1078 H=1078:00 +CPU= 44.359
  19:16:57 STEP 1079 H=1079:00 +CPU= 44.599
  19:18:22 STEP 1080 H=1080:00 +CPU= 79.840
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001080
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001080
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001056
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001068
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001068
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001068
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_44.zip
Uploading the intermediate file: upload_file_44.zip
  19:19:07 STEP 1081 H=1081:00 +CPU= 43.082
  19:19:53 STEP 1082 H=1082:00 +CPU= 44.503
  19:21:05 STEP 1083 H=1083:00 +CPU= 67.921
  19:21:51 STEP 1084 H=1084:00 +CPU= 44.444
  19:22:36 STEP 1085 H=1085:00 +CPU= 43.071
  19:23:49 STEP 1086 H=1086:00 +CPU= 69.292
  19:24:34 STEP 1087 H=1087:00 +CPU= 43.242
  19:25:20 STEP 1088 H=1088:00 +CPU= 44.488
  19:26:31 STEP 1089 H=1089:00 +CPU= 67.899
  19:27:18 STEP 1090 H=1090:00 +CPU= 44.554
  19:28:03 STEP 1091 H=1091:00 +CPU= 43.188
  19:29:22 STEP 1092 H=1092:00 +CPU= 75.795
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001092
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001092
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001092
  19:30:08 STEP 1093 H=1093:00 +CPU= 43.175
  19:30:54 STEP 1094 H=1094:00 +CPU= 44.698
  19:32:06 STEP 1095 H=1095:00 +CPU= 68.180
  19:32:52 STEP 1096 H=1096:00 +CPU= 44.398
  19:33:38 STEP 1097 H=1097:00 +CPU= 43.344
  19:34:50 STEP 1098 H=1098:00 +CPU= 69.238
  19:35:35 STEP 1099 H=1099:00 +CPU= 43.096
  19:36:22 STEP 1100 H=1100:00 +CPU= 44.395
  19:37:32 STEP 1101 H=1101:00 +CPU= 67.712
  19:38:19 STEP 1102 H=1102:00 +CPU= 44.412
  19:39:06 STEP 1103 H=1103:00 +CPU= 44.643
  19:40:31 STEP 1104 H=1104:00 +CPU= 79.926
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001104
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001104
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001080
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001092
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001092
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001092
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_45.zip
Uploading the intermediate file: upload_file_45.zip
  19:41:16 STEP 1105 H=1105:00 +CPU= 43.172
  19:42:02 STEP 1106 H=1106:00 +CPU= 44.385
  19:43:13 STEP 1107 H=1107:00 +CPU= 67.743
  19:44:00 STEP 1108 H=1108:00 +CPU= 44.415
  19:44:45 STEP 1109 H=1109:00 +CPU= 43.144
  19:45:57 STEP 1110 H=1110:00 +CPU= 69.300
  19:46:43 STEP 1111 H=1111:00 +CPU= 43.242
  19:47:29 STEP 1112 H=1112:00 +CPU= 44.447
  19:48:40 STEP 1113 H=1113:00 +CPU= 67.952
  19:49:27 STEP 1114 H=1114:00 +CPU= 44.411
  19:50:12 STEP 1115 H=1115:00 +CPU= 43.215
  19:51:31 STEP 1116 H=1116:00 +CPU= 75.858
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001116
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001116
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001116
  19:52:17 STEP 1117 H=1117:00 +CPU= 43.208
  19:53:03 STEP 1118 H=1118:00 +CPU= 44.619
  19:54:14 STEP 1119 H=1119:00 +CPU= 67.926
  19:55:01 STEP 1120 H=1120:00 +CPU= 44.569
  19:55:46 STEP 1121 H=1121:00 +CPU= 43.253
  19:56:59 STEP 1122 H=1122:00 +CPU= 69.276
  19:57:44 STEP 1123 H=1123:00 +CPU= 43.060
  19:58:30 STEP 1124 H=1124:00 +CPU= 44.287
  19:59:41 STEP 1125 H=1125:00 +CPU= 67.631
  20:00:27 STEP 1126 H=1126:00 +CPU= 44.420
  20:01:14 STEP 1127 H=1127:00 +CPU= 44.546
  20:02:39 STEP 1128 H=1128:00 +CPU= 79.876
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001128
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001128
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001104
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001116
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001116
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001116
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_46.zip
Uploading the intermediate file: upload_file_46.zip
  20:03:24 STEP 1129 H=1129:00 +CPU= 43.059
  20:04:11 STEP 1130 H=1130:00 +CPU= 44.341
  20:05:21 STEP 1131 H=1131:00 +CPU= 67.786
  20:06:08 STEP 1132 H=1132:00 +CPU= 44.349
  20:06:53 STEP 1133 H=1133:00 +CPU= 43.199
  20:08:05 STEP 1134 H=1134:00 +CPU= 69.201
  20:08:51 STEP 1135 H=1135:00 +CPU= 43.232
  20:09:37 STEP 1136 H=1136:00 +CPU= 44.440
  20:10:48 STEP 1137 H=1137:00 +CPU= 67.731
  20:11:34 STEP 1138 H=1138:00 +CPU= 44.429
  20:12:20 STEP 1139 H=1139:00 +CPU= 43.185
  20:13:39 STEP 1140 H=1140:00 +CPU= 75.792
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001140
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001140
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001140
  20:14:24 STEP 1141 H=1141:00 +CPU= 43.261
  20:15:11 STEP 1142 H=1142:00 +CPU= 44.524
  20:16:22 STEP 1143 H=1143:00 +CPU= 67.924
  20:17:09 STEP 1144 H=1144:00 +CPU= 44.429
  20:17:54 STEP 1145 H=1145:00 +CPU= 43.025
  20:19:06 STEP 1146 H=1146:00 +CPU= 69.093
  20:19:51 STEP 1147 H=1147:00 +CPU= 43.207
  20:20:38 STEP 1148 H=1148:00 +CPU= 44.363
  20:21:48 STEP 1149 H=1149:00 +CPU= 67.770
  20:22:35 STEP 1150 H=1150:00 +CPU= 44.355
  20:23:21 STEP 1151 H=1151:00 +CPU= 44.505
  20:24:46 STEP 1152 H=1152:00 +CPU= 79.657
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001152
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001152
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001128
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001140
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001140
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001140
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_47.zip
Uploading the intermediate file: upload_file_47.zip
  20:25:31 STEP 1153 H=1153:00 +CPU= 42.989
  20:26:18 STEP 1154 H=1154:00 +CPU= 44.427
  20:27:29 STEP 1155 H=1155:00 +CPU= 67.888
  20:28:16 STEP 1156 H=1156:00 +CPU= 44.649
  20:29:01 STEP 1157 H=1157:00 +CPU= 43.145
  20:30:13 STEP 1158 H=1158:00 +CPU= 69.127
  20:30:59 STEP 1159 H=1159:00 +CPU= 43.275
  20:31:45 STEP 1160 H=1160:00 +CPU= 44.635
  20:32:56 STEP 1161 H=1161:00 +CPU= 67.918
  20:33:43 STEP 1162 H=1162:00 +CPU= 44.529
  20:34:28 STEP 1163 H=1163:00 +CPU= 43.182
  20:35:48 STEP 1164 H=1164:00 +CPU= 75.952
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001164
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001164
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001164
  20:36:33 STEP 1165 H=1165:00 +CPU= 43.249
  20:37:20 STEP 1166 H=1166:00 +CPU= 44.442
  20:38:31 STEP 1167 H=1167:00 +CPU= 67.953
  20:39:17 STEP 1168 H=1168:00 +CPU= 44.433
  20:40:03 STEP 1169 H=1169:00 +CPU= 43.235
  20:41:15 STEP 1170 H=1170:00 +CPU= 69.236
  20:42:00 STEP 1171 H=1171:00 +CPU= 43.210
  20:42:47 STEP 1172 H=1172:00 +CPU= 44.426
  20:43:58 STEP 1173 H=1173:00 +CPU= 67.736
  20:44:44 STEP 1174 H=1174:00 +CPU= 44.293
  20:45:31 STEP 1175 H=1175:00 +CPU= 44.532
  20:46:55 STEP 1176 H=1176:00 +CPU= 79.651
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001176
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001176
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001152
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001164
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001164
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001164
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_48.zip
Uploading the intermediate file: upload_file_48.zip
  20:47:41 STEP 1177 H=1177:00 +CPU= 43.119
  20:48:27 STEP 1178 H=1178:00 +CPU= 44.324
  20:49:38 STEP 1179 H=1179:00 +CPU= 67.931
  20:50:25 STEP 1180 H=1180:00 +CPU= 44.494
  20:51:10 STEP 1181 H=1181:00 +CPU= 43.245
  20:52:23 STEP 1182 H=1182:00 +CPU= 69.327
  20:53:08 STEP 1183 H=1183:00 +CPU= 43.219
  20:53:54 STEP 1184 H=1184:00 +CPU= 44.441
  20:55:06 STEP 1185 H=1185:00 +CPU= 67.912
  20:55:52 STEP 1186 H=1186:00 +CPU= 44.564
  20:56:37 STEP 1187 H=1187:00 +CPU= 43.215
  20:57:57 STEP 1188 H=1188:00 +CPU= 75.938
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001188
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001188
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001188
  20:58:42 STEP 1189 H=1189:00 +CPU= 43.158
  20:59:29 STEP 1190 H=1190:00 +CPU= 44.536
  21:00:40 STEP 1191 H=1191:00 +CPU= 67.845
  21:01:26 STEP 1192 H=1192:00 +CPU= 44.458
  21:02:12 STEP 1193 H=1193:00 +CPU= 43.177
  21:03:24 STEP 1194 H=1194:00 +CPU= 69.241
  21:04:09 STEP 1195 H=1195:00 +CPU= 42.935
  21:04:56 STEP 1196 H=1196:00 +CPU= 44.489
  21:06:07 STEP 1197 H=1197:00 +CPU= 67.888
  21:06:54 STEP 1198 H=1198:00 +CPU= 44.392
  21:07:40 STEP 1199 H=1199:00 +CPU= 44.451
  21:09:05 STEP 1200 H=1200:00 +CPU= 79.576
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001200
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001200
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001176
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001188
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001188
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001188
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_49.zip
Uploading the intermediate file: upload_file_49.zip
Uploading trickle at timestep: 4320000
  21:09:50 STEP 1201 H=1201:00 +CPU= 43.168
  21:10:37 STEP 1202 H=1202:00 +CPU= 44.513
  21:11:47 STEP 1203 H=1203:00 +CPU= 67.665
  21:12:34 STEP 1204 H=1204:00 +CPU= 44.350
  21:13:19 STEP 1205 H=1205:00 +CPU= 43.115
  21:14:31 STEP 1206 H=1206:00 +CPU= 69.100
  21:15:17 STEP 1207 H=1207:00 +CPU= 43.302
  21:16:03 STEP 1208 H=1208:00 +CPU= 44.462
  21:17:14 STEP 1209 H=1209:00 +CPU= 67.751
  21:18:01 STEP 1210 H=1210:00 +CPU= 44.517
  21:18:46 STEP 1211 H=1211:00 +CPU= 43.141
  21:20:05 STEP 1212 H=1212:00 +CPU= 76.146
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001212
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001212
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001212
  21:20:51 STEP 1213 H=1213:00 +CPU= 43.246
  21:21:37 STEP 1214 H=1214:00 +CPU= 44.428
  21:22:48 STEP 1215 H=1215:00 +CPU= 67.595
  21:23:34 STEP 1216 H=1216:00 +CPU= 44.486
  21:24:19 STEP 1217 H=1217:00 +CPU= 43.083
  21:25:32 STEP 1218 H=1218:00 +CPU= 69.322
  21:26:17 STEP 1219 H=1219:00 +CPU= 43.213
  21:27:04 STEP 1220 H=1220:00 +CPU= 44.401
  21:28:15 STEP 1221 H=1221:00 +CPU= 67.721
  21:29:01 STEP 1222 H=1222:00 +CPU= 44.327
  21:29:48 STEP 1223 H=1223:00 +CPU= 44.494
  21:31:12 STEP 1224 H=1224:00 +CPU= 79.677
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001224
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001224
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001200
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001212
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001212
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001212
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_50.zip
Uploading the intermediate file: upload_file_50.zip
  21:31:57 STEP 1225 H=1225:00 +CPU= 42.963
  21:32:43 STEP 1226 H=1226:00 +CPU= 44.303
  21:33:55 STEP 1227 H=1227:00 +CPU= 67.862
  21:34:41 STEP 1228 H=1228:00 +CPU= 44.479
  21:35:26 STEP 1229 H=1229:00 +CPU= 43.166
  21:36:39 STEP 1230 H=1230:00 +CPU= 69.415
  21:37:24 STEP 1231 H=1231:00 +CPU= 43.134
  21:38:11 STEP 1232 H=1232:00 +CPU= 44.475
  21:39:22 STEP 1233 H=1233:00 +CPU= 67.789
  21:40:08 STEP 1234 H=1234:00 +CPU= 44.505
  21:40:53 STEP 1235 H=1235:00 +CPU= 43.291
  21:42:13 STEP 1236 H=1236:00 +CPU= 75.853
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001236
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001236
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001236
  21:42:58 STEP 1237 H=1237:00 +CPU= 43.221
  21:43:45 STEP 1238 H=1238:00 +CPU= 44.630
  21:44:56 STEP 1239 H=1239:00 +CPU= 67.787
  21:45:42 STEP 1240 H=1240:00 +CPU= 44.369
  21:46:27 STEP 1241 H=1241:00 +CPU= 43.069
  21:47:40 STEP 1242 H=1242:00 +CPU= 69.266
  21:48:25 STEP 1243 H=1243:00 +CPU= 43.153
  21:49:12 STEP 1244 H=1244:00 +CPU= 44.285
  21:50:22 STEP 1245 H=1245:00 +CPU= 67.610
  21:51:09 STEP 1246 H=1246:00 +CPU= 44.295
  21:51:55 STEP 1247 H=1247:00 +CPU= 44.565
  21:53:20 STEP 1248 H=1248:00 +CPU= 79.702
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001248
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001248
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001224
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001236
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001236
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001236
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_51.zip
Uploading the intermediate file: upload_file_51.zip
  21:54:05 STEP 1249 H=1249:00 +CPU= 43.021
  21:54:51 STEP 1250 H=1250:00 +CPU= 44.449
  21:56:03 STEP 1251 H=1251:00 +CPU= 67.849
  21:56:49 STEP 1252 H=1252:00 +CPU= 44.468
  21:57:34 STEP 1253 H=1253:00 +CPU= 43.063
  21:58:47 STEP 1254 H=1254:00 +CPU= 69.285
  21:59:32 STEP 1255 H=1255:00 +CPU= 43.092
  22:00:18 STEP 1256 H=1256:00 +CPU= 44.496
  22:01:30 STEP 1257 H=1257:00 +CPU= 67.933
  22:02:16 STEP 1258 H=1258:00 +CPU= 44.458
  22:03:02 STEP 1259 H=1259:00 +CPU= 43.290
  22:04:21 STEP 1260 H=1260:00 +CPU= 75.733
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001260
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001260
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001260
  22:05:07 STEP 1261 H=1261:00 +CPU= 43.398
  22:05:53 STEP 1262 H=1262:00 +CPU= 44.381
  22:07:04 STEP 1263 H=1263:00 +CPU= 67.787
  22:07:51 STEP 1264 H=1264:00 +CPU= 44.541
  22:08:36 STEP 1265 H=1265:00 +CPU= 43.314
  22:09:48 STEP 1266 H=1266:00 +CPU= 69.148
  22:10:34 STEP 1267 H=1267:00 +CPU= 43.040
  22:11:20 STEP 1268 H=1268:00 +CPU= 44.312
  22:12:31 STEP 1269 H=1269:00 +CPU= 67.760
  22:13:18 STEP 1270 H=1270:00 +CPU= 44.437
  22:14:05 STEP 1271 H=1271:00 +CPU= 44.536
  22:15:29 STEP 1272 H=1272:00 +CPU= 79.781
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001272
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001272
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001248
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001260
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001260
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001260
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_52.zip
Uploading the intermediate file: upload_file_52.zip
  22:16:14 STEP 1273 H=1273:00 +CPU= 43.104
  22:17:01 STEP 1274 H=1274:00 +CPU= 44.416
  22:18:12 STEP 1275 H=1275:00 +CPU= 67.724
  22:18:58 STEP 1276 H=1276:00 +CPU= 44.446
  22:19:43 STEP 1277 H=1277:00 +CPU= 43.178
  22:20:56 STEP 1278 H=1278:00 +CPU= 69.207
  22:21:41 STEP 1279 H=1279:00 +CPU= 43.162
  22:22:28 STEP 1280 H=1280:00 +CPU= 44.514
  22:23:39 STEP 1281 H=1281:00 +CPU= 67.967
  22:24:25 STEP 1282 H=1282:00 +CPU= 44.389
  22:25:10 STEP 1283 H=1283:00 +CPU= 43.307
  22:26:30 STEP 1284 H=1284:00 +CPU= 75.833
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001284
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001284
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001284
  22:27:15 STEP 1285 H=1285:00 +CPU= 43.174
  22:28:02 STEP 1286 H=1286:00 +CPU= 44.595
  22:29:13 STEP 1287 H=1287:00 +CPU= 67.814
  22:29:59 STEP 1288 H=1288:00 +CPU= 44.472
  22:30:44 STEP 1289 H=1289:00 +CPU= 43.222
  22:31:57 STEP 1290 H=1290:00 +CPU= 69.178
  22:32:42 STEP 1291 H=1291:00 +CPU= 43.139
  22:33:28 STEP 1292 H=1292:00 +CPU= 44.390
  22:34:39 STEP 1293 H=1293:00 +CPU= 67.597
  22:35:26 STEP 1294 H=1294:00 +CPU= 44.470
  22:36:12 STEP 1295 H=1295:00 +CPU= 44.690
  22:37:37 STEP 1296 H=1296:00 +CPU= 79.665
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001296
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001296
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001272
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001284
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001284
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001284
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_53.zip
Uploading the intermediate file: upload_file_53.zip
  22:38:22 STEP 1297 H=1297:00 +CPU= 43.123
  22:39:09 STEP 1298 H=1298:00 +CPU= 44.513
  22:40:19 STEP 1299 H=1299:00 +CPU= 67.612
  22:41:06 STEP 1300 H=1300:00 +CPU= 44.391
  22:41:51 STEP 1301 H=1301:00 +CPU= 43.104
  22:43:03 STEP 1302 H=1302:00 +CPU= 69.039
  22:43:48 STEP 1303 H=1303:00 +CPU= 43.155
  22:44:35 STEP 1304 H=1304:00 +CPU= 44.243
  22:45:46 STEP 1305 H=1305:00 +CPU= 67.843
  22:46:32 STEP 1306 H=1306:00 +CPU= 44.441
  22:47:18 STEP 1307 H=1307:00 +CPU= 43.283
  22:48:37 STEP 1308 H=1308:00 +CPU= 75.644
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001308
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001308
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001308
  22:49:22 STEP 1309 H=1309:00 +CPU= 43.216
  22:50:09 STEP 1310 H=1310:00 +CPU= 44.518
  22:51:20 STEP 1311 H=1311:00 +CPU= 67.907
  22:52:07 STEP 1312 H=1312:00 +CPU= 44.363
  22:52:52 STEP 1313 H=1313:00 +CPU= 43.288
  22:54:05 STEP 1314 H=1314:00 +CPU= 69.456
  22:54:50 STEP 1315 H=1315:00 +CPU= 43.168
  22:55:36 STEP 1316 H=1316:00 +CPU= 44.439
  22:56:47 STEP 1317 H=1317:00 +CPU= 67.720
  22:57:34 STEP 1318 H=1318:00 +CPU= 44.361
  22:58:20 STEP 1319 H=1319:00 +CPU= 44.443
  22:59:45 STEP 1320 H=1320:00 +CPU= 79.746
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001320
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001320
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001296
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001308
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001308
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001308
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_54.zip
Uploading the intermediate file: upload_file_54.zip
  23:00:30 STEP 1321 H=1321:00 +CPU= 43.195
  23:01:17 STEP 1322 H=1322:00 +CPU= 44.328
  23:02:28 STEP 1323 H=1323:00 +CPU= 67.841
  23:03:14 STEP 1324 H=1324:00 +CPU= 44.395
  23:03:59 STEP 1325 H=1325:00 +CPU= 42.928
  23:05:11 STEP 1326 H=1326:00 +CPU= 69.268
  23:05:57 STEP 1327 H=1327:00 +CPU= 43.082
  23:06:43 STEP 1328 H=1328:00 +CPU= 44.470
  23:07:54 STEP 1329 H=1329:00 +CPU= 67.793
  23:08:41 STEP 1330 H=1330:00 +CPU= 44.369
  23:09:26 STEP 1331 H=1331:00 +CPU= 43.308
  23:10:45 STEP 1332 H=1332:00 +CPU= 75.924
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001332
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001332
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001332
  23:11:31 STEP 1333 H=1333:00 +CPU= 43.154
  23:12:17 STEP 1334 H=1334:00 +CPU= 44.490
  23:13:28 STEP 1335 H=1335:00 +CPU= 67.889
  23:14:15 STEP 1336 H=1336:00 +CPU= 44.366
  23:15:00 STEP 1337 H=1337:00 +CPU= 43.136
  23:16:13 STEP 1338 H=1338:00 +CPU= 69.412
  23:16:58 STEP 1339 H=1339:00 +CPU= 43.094
  23:17:44 STEP 1340 H=1340:00 +CPU= 44.432
  23:18:55 STEP 1341 H=1341:00 +CPU= 67.775
  23:19:42 STEP 1342 H=1342:00 +CPU= 44.234
  23:20:28 STEP 1343 H=1343:00 +CPU= 44.472
  23:21:52 STEP 1344 H=1344:00 +CPU= 79.660
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001344
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001344
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001320
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001332
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001332
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001332
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_55.zip
Uploading the intermediate file: upload_file_55.zip
  23:22:38 STEP 1345 H=1345:00 +CPU= 43.033
  23:23:24 STEP 1346 H=1346:00 +CPU= 44.392
  23:24:35 STEP 1347 H=1347:00 +CPU= 67.471
  23:25:21 STEP 1348 H=1348:00 +CPU= 44.509
  23:26:06 STEP 1349 H=1349:00 +CPU= 43.063
  23:27:18 STEP 1350 H=1350:00 +CPU= 69.055
  23:28:04 STEP 1351 H=1351:00 +CPU= 43.109
  23:28:50 STEP 1352 H=1352:00 +CPU= 44.441
  23:30:01 STEP 1353 H=1353:00 +CPU= 67.688
  23:30:48 STEP 1354 H=1354:00 +CPU= 44.366
  23:31:33 STEP 1355 H=1355:00 +CPU= 43.140
  23:32:52 STEP 1356 H=1356:00 +CPU= 75.709
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001356
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001356
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001356
  23:33:37 STEP 1357 H=1357:00 +CPU= 43.167
  23:34:24 STEP 1358 H=1358:00 +CPU= 44.579
  23:35:35 STEP 1359 H=1359:00 +CPU= 67.733
  23:36:21 STEP 1360 H=1360:00 +CPU= 44.467
  23:37:06 STEP 1361 H=1361:00 +CPU= 43.080
  23:38:19 STEP 1362 H=1362:00 +CPU= 69.130
  23:39:04 STEP 1363 H=1363:00 +CPU= 43.153
  23:39:50 STEP 1364 H=1364:00 +CPU= 44.420
  23:41:02 STEP 1365 H=1365:00 +CPU= 68.015
  23:41:48 STEP 1366 H=1366:00 +CPU= 44.382
  23:42:35 STEP 1367 H=1367:00 +CPU= 44.457
  23:43:59 STEP 1368 H=1368:00 +CPU= 79.667
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001368
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001368
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001344
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001356
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001356
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001356
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_56.zip
Uploading the intermediate file: upload_file_56.zip
  23:44:44 STEP 1369 H=1369:00 +CPU= 43.020
  23:45:31 STEP 1370 H=1370:00 +CPU= 44.474
  23:46:42 STEP 1371 H=1371:00 +CPU= 67.639
  23:47:29 STEP 1372 H=1372:00 +CPU= 44.422
  23:48:14 STEP 1373 H=1373:00 +CPU= 43.088
  23:49:27 STEP 1374 H=1374:00 +CPU= 69.240
  23:50:12 STEP 1375 H=1375:00 +CPU= 43.025
  23:50:58 STEP 1376 H=1376:00 +CPU= 44.472
  23:52:09 STEP 1377 H=1377:00 +CPU= 67.690
  23:52:55 STEP 1378 H=1378:00 +CPU= 44.420
  23:53:41 STEP 1379 H=1379:00 +CPU= 43.152
  23:55:00 STEP 1380 H=1380:00 +CPU= 75.798
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001380
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001380
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001380
  23:55:45 STEP 1381 H=1381:00 +CPU= 43.129
  23:56:32 STEP 1382 H=1382:00 +CPU= 44.394
  23:57:42 STEP 1383 H=1383:00 +CPU= 67.692
  23:58:29 STEP 1384 H=1384:00 +CPU= 44.371
  23:59:14 STEP 1385 H=1385:00 +CPU= 43.069
  00:00:27 STEP 1386 H=1386:00 +CPU= 69.233
  00:01:12 STEP 1387 H=1387:00 +CPU= 43.091
  00:01:58 STEP 1388 H=1388:00 +CPU= 44.429
  00:03:09 STEP 1389 H=1389:00 +CPU= 67.690
  00:03:55 STEP 1390 H=1390:00 +CPU= 44.282
  00:04:42 STEP 1391 H=1391:00 +CPU= 44.487
  00:06:07 STEP 1392 H=1392:00 +CPU= 79.644
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001392
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001392
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001368
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001380
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001380
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001380
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_57.zip
Uploading the intermediate file: upload_file_57.zip
  00:06:52 STEP 1393 H=1393:00 +CPU= 43.084
  00:07:38 STEP 1394 H=1394:00 +CPU= 44.218
  00:08:49 STEP 1395 H=1395:00 +CPU= 67.599
  00:09:35 STEP 1396 H=1396:00 +CPU= 44.424
  00:10:20 STEP 1397 H=1397:00 +CPU= 43.116
  00:11:33 STEP 1398 H=1398:00 +CPU= 69.104
  00:12:18 STEP 1399 H=1399:00 +CPU= 43.045
  00:13:04 STEP 1400 H=1400:00 +CPU= 44.353
  00:14:15 STEP 1401 H=1401:00 +CPU= 67.861
  00:15:02 STEP 1402 H=1402:00 +CPU= 44.481
  00:15:47 STEP 1403 H=1403:00 +CPU= 43.202
  00:17:06 STEP 1404 H=1404:00 +CPU= 75.720
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001404
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001404
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001404
  00:17:51 STEP 1405 H=1405:00 +CPU= 43.063
  00:18:38 STEP 1406 H=1406:00 +CPU= 44.434
  00:19:49 STEP 1407 H=1407:00 +CPU= 67.652
  00:20:36 STEP 1408 H=1408:00 +CPU= 44.499
  00:21:21 STEP 1409 H=1409:00 +CPU= 43.161
  00:22:33 STEP 1410 H=1410:00 +CPU= 69.177
  00:23:18 STEP 1411 H=1411:00 +CPU= 43.184
  00:24:05 STEP 1412 H=1412:00 +CPU= 44.223
  00:25:16 STEP 1413 H=1413:00 +CPU= 67.874
  00:26:03 STEP 1414 H=1414:00 +CPU= 44.575
  00:26:49 STEP 1415 H=1415:00 +CPU= 44.591
  00:28:14 STEP 1416 H=1416:00 +CPU= 79.941
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001416
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001416
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001392
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001404
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001404
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001404
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_58.zip
Uploading the intermediate file: upload_file_58.zip
  00:28:59 STEP 1417 H=1417:00 +CPU= 42.977
  00:29:45 STEP 1418 H=1418:00 +CPU= 44.380
  00:30:56 STEP 1419 H=1419:00 +CPU= 67.708
  00:31:43 STEP 1420 H=1420:00 +CPU= 44.197
  00:32:28 STEP 1421 H=1421:00 +CPU= 43.165
  00:33:40 STEP 1422 H=1422:00 +CPU= 69.294
  00:34:25 STEP 1423 H=1423:00 +CPU= 43.204
  00:35:12 STEP 1424 H=1424:00 +CPU= 44.413
  00:36:23 STEP 1425 H=1425:00 +CPU= 67.620
  00:37:09 STEP 1426 H=1426:00 +CPU= 44.291
  00:37:54 STEP 1427 H=1427:00 +CPU= 43.185
  00:39:13 STEP 1428 H=1428:00 +CPU= 75.688
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001428
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001428
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001428
  00:39:58 STEP 1429 H=1429:00 +CPU= 43.063
  00:40:45 STEP 1430 H=1430:00 +CPU= 44.377
  00:41:56 STEP 1431 H=1431:00 +CPU= 67.752
  00:42:43 STEP 1432 H=1432:00 +CPU= 44.515
  00:43:28 STEP 1433 H=1433:00 +CPU= 43.121
  00:44:40 STEP 1434 H=1434:00 +CPU= 69.087
  00:45:25 STEP 1435 H=1435:00 +CPU= 43.084
  00:46:12 STEP 1436 H=1436:00 +CPU= 44.400
  00:47:23 STEP 1437 H=1437:00 +CPU= 67.683
  00:48:09 STEP 1438 H=1438:00 +CPU= 44.220
  00:48:56 STEP 1439 H=1439:00 +CPU= 44.708
  00:50:20 STEP 1440 H=1440:00 +CPU= 79.735
Moving to projects directory: /home/luuser/slots/1/ICMGGhq0f+001440
Moving to projects directory: /home/luuser/slots/1/ICMSHhq0f+001440
Moving to projects directory: /home/luuser/slots/1/ICMUAhq0f+001440
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001416
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMGGhq0f+001428
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMSHhq0f+001428
Adding to the zip: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_12205364/ICMUAhq0f+001428
Zipping up the intermediate file: /home/luuser/projects/climateprediction.net/oifs_43r3_ps_0720_2019050100_123_988_12205364_1_r1225456425_59.zip
Uploading the intermediate file: upload_file_59.zip
Uploading trickle at timestep: 5184000
  00:51:07 STEP 1441 H=1441:00 +CPU= 43.263
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:39:56 1537854 22302631 oifs_43r3_ps_0720_2019050100_123_988_12205364_1 5,184,000 76,191 0.0147
09 May 2023 16:39:43 1537854 22302631 oifs_43r3_ps_0720_2019050100_123_988_12205364_1 4,320,000 63,523 0.0147
09 May 2023 16:39:30 1537854 22302631 oifs_43r3_ps_0720_2019050100_123_988_12205364_1 3,456,000 50,841 0.0147
09 May 2023 16:39:15 1537854 22302631 oifs_43r3_ps_0720_2019050100_123_988_12205364_1 2,592,000 38,152 0.0147
09 May 2023 16:39:01 1537854 22302631 oifs_43r3_ps_0720_2019050100_123_988_12205364_1 1,728,000 25,468 0.0147
09 May 2023 16:38:47 1537854 22302631 oifs_43r3_ps_0720_2019050100_123_988_12205364_1 864,000 12,780 0.0148


©2024 cpdn.org