climateprediction.net home page
Task 22440419

Task 22440419

Name oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1
Workunit 12284957
Created 8 Jun 2024, 4:40:05 UTC
Sent 8 Jun 2024, 4:52:30 UTC
Report deadline 7 Aug 2024, 4:52:30 UTC
Received 9 Jun 2024, 3:14:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1536520
Run time 21 hours 23 min 52 sec
CPU time 20 hours 56 min 46 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.37 MB
Peak swap size 5,980.80 MB
Peak disk usage 2,223.44 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
s/12/ICMUAh7zg+000780
  12:33:37 STEP  781 H= 195:15 +CPU= 54.999
  12:34:32 STEP  782 H= 195:30 +CPU= 54.246
  12:35:27 STEP  783 H= 195:45 +CPU= 54.047
  12:36:23 STEP  784 H= 196:00 +CPU= 55.967
  12:37:20 STEP  785 H= 196:15 +CPU= 55.814
  12:38:15 STEP  786 H= 196:30 +CPU= 54.581
  12:39:11 STEP  787 H= 196:45 +CPU= 55.979
  12:40:07 STEP  788 H= 197:00 +CPU= 55.907
  12:41:02 STEP  789 H= 197:15 +CPU= 53.746
  12:41:58 STEP  790 H= 197:30 +CPU= 55.142
  12:42:53 STEP  791 H= 197:45 +CPU= 54.478
  12:44:00 STEP  792 H= 198:00 +CPU= 66.151
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000792
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000792
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000792
  12:44:55 STEP  793 H= 198:15 +CPU= 54.924
  12:45:50 STEP  794 H= 198:30 +CPU= 54.192
  12:46:45 STEP  795 H= 198:45 +CPU= 53.781
  12:47:41 STEP  796 H= 199:00 +CPU= 55.168
  12:48:36 STEP  797 H= 199:15 +CPU= 54.178
  12:49:30 STEP  798 H= 199:30 +CPU= 53.242
  12:50:25 STEP  799 H= 199:45 +CPU= 54.239
  12:51:14 STEP  800 H= 200:00 +CPU= 49.032
  12:52:04 STEP  801 H= 200:15 +CPU= 48.703
  12:52:53 STEP  802 H= 200:30 +CPU= 48.707
  12:53:42 STEP  803 H= 200:45 +CPU= 48.726
  12:54:27 STEP  804 H= 201:00 +CPU= 44.880
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000804
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000804
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000804
  12:55:04 STEP  805 H= 201:15 +CPU= 36.736
  12:55:40 STEP  806 H= 201:30 +CPU= 36.161
  12:56:17 STEP  807 H= 201:45 +CPU= 36.511
  12:56:55 STEP  808 H= 202:00 +CPU= 37.794
  12:57:37 STEP  809 H= 202:15 +CPU= 40.869
  12:58:22 STEP  810 H= 202:30 +CPU= 45.012
  12:59:08 STEP  811 H= 202:45 +CPU= 45.521
  12:59:54 STEP  812 H= 203:00 +CPU= 45.326
  13:00:40 STEP  813 H= 203:15 +CPU= 46.102
  13:01:26 STEP  814 H= 203:30 +CPU= 45.686
  13:02:12 STEP  815 H= 203:45 +CPU= 45.192
  13:03:16 STEP  816 H= 204:00 +CPU= 63.061
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000816
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000816
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000816
  13:04:02 STEP  817 H= 204:15 +CPU= 45.686
  13:04:47 STEP  818 H= 204:30 +CPU= 45.320
  13:05:33 STEP  819 H= 204:45 +CPU= 45.670
  13:06:18 STEP  820 H= 205:00 +CPU= 44.527
  13:07:04 STEP  821 H= 205:15 +CPU= 45.608
  13:07:50 STEP  822 H= 205:30 +CPU= 45.353
  13:08:35 STEP  823 H= 205:45 +CPU= 44.654
  13:09:21 STEP  824 H= 206:00 +CPU= 45.159
  13:10:07 STEP  825 H= 206:15 +CPU= 45.623
  13:10:52 STEP  826 H= 206:30 +CPU= 44.681
  13:11:38 STEP  827 H= 206:45 +CPU= 45.291
  13:12:34 STEP  828 H= 207:00 +CPU= 56.046
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000828
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000828
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000828
  13:13:20 STEP  829 H= 207:15 +CPU= 45.507
  13:14:06 STEP  830 H= 207:30 +CPU= 45.364
  13:14:51 STEP  831 H= 207:45 +CPU= 45.203
  13:15:39 STEP  832 H= 208:00 +CPU= 47.406
  13:16:25 STEP  833 H= 208:15 +CPU= 45.245
  13:17:10 STEP  834 H= 208:30 +CPU= 45.415
  13:17:56 STEP  835 H= 208:45 +CPU= 44.607
  13:18:44 STEP  836 H= 209:00 +CPU= 45.673
  13:19:33 STEP  837 H= 209:15 +CPU= 46.190
  13:20:20 STEP  838 H= 209:30 +CPU= 45.764
  13:21:07 STEP  839 H= 209:45 +CPU= 45.853
  13:22:05 STEP  840 H= 210:00 +CPU= 55.863
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000840
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000840
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000840
  13:22:52 STEP  841 H= 210:15 +CPU= 44.831
  13:23:43 STEP  842 H= 210:30 +CPU= 44.685
  13:24:31 STEP  843 H= 210:45 +CPU= 42.533
  13:25:20 STEP  844 H= 211:00 +CPU= 45.940
  13:26:10 STEP  845 H= 211:15 +CPU= 45.319
  13:26:57 STEP  846 H= 211:30 +CPU= 45.427
  13:27:44 STEP  847 H= 211:45 +CPU= 44.490
  13:28:33 STEP  848 H= 212:00 +CPU= 45.787
  13:29:19 STEP  849 H= 212:15 +CPU= 44.265
  13:30:05 STEP  850 H= 212:30 +CPU= 45.306
  13:30:50 STEP  851 H= 212:45 +CPU= 45.138
  13:31:47 STEP  852 H= 213:00 +CPU= 56.039
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000852
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000852
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000852
  13:32:33 STEP  853 H= 213:15 +CPU= 45.244
  13:33:19 STEP  854 H= 213:30 +CPU= 45.338
  13:34:04 STEP  855 H= 213:45 +CPU= 45.181
  13:34:52 STEP  856 H= 214:00 +CPU= 45.524
  13:35:43 STEP  857 H= 214:15 +CPU= 45.767
  13:36:30 STEP  858 H= 214:30 +CPU= 45.074
  13:37:17 STEP  859 H= 214:45 +CPU= 44.451
  13:38:06 STEP  860 H= 215:00 +CPU= 46.891
  13:38:55 STEP  861 H= 215:15 +CPU= 45.231
  13:39:40 STEP  862 H= 215:30 +CPU= 44.555
  13:40:27 STEP  863 H= 215:45 +CPU= 45.726
  13:41:30 STEP  864 H= 216:00 +CPU= 61.182
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000864
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000864
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_8.zip
Uploading the intermediate file: upload_file_8.zip
  13:42:16 STEP  865 H= 216:15 +CPU= 44.865
  13:43:03 STEP  866 H= 216:30 +CPU= 45.250
  13:43:51 STEP  867 H= 216:45 +CPU= 45.751
  13:44:39 STEP  868 H= 217:00 +CPU= 45.977
  13:45:27 STEP  869 H= 217:15 +CPU= 45.570
  13:46:17 STEP  870 H= 217:30 +CPU= 45.321
  13:47:05 STEP  871 H= 217:45 +CPU= 45.450
  13:47:54 STEP  872 H= 218:00 +CPU= 47.104
  13:48:43 STEP  873 H= 218:15 +CPU= 46.166
  13:49:32 STEP  874 H= 218:30 +CPU= 45.153
  13:50:19 STEP  875 H= 218:45 +CPU= 45.245
  13:51:18 STEP  876 H= 219:00 +CPU= 55.581
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000876
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000876
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000876
  13:52:02 STEP  877 H= 219:15 +CPU= 44.304
  13:52:48 STEP  878 H= 219:30 +CPU= 45.022
  13:53:33 STEP  879 H= 219:45 +CPU= 44.905
  13:54:20 STEP  880 H= 220:00 +CPU= 46.264
  13:55:04 STEP  881 H= 220:15 +CPU= 44.192
  13:55:50 STEP  882 H= 220:30 +CPU= 45.063
  13:56:36 STEP  883 H= 220:45 +CPU= 46.165
  13:57:23 STEP  884 H= 221:00 +CPU= 46.186
  13:58:09 STEP  885 H= 221:15 +CPU= 45.131
  13:58:55 STEP  886 H= 221:30 +CPU= 45.834
  13:59:40 STEP  887 H= 221:45 +CPU= 45.221
  14:00:37 STEP  888 H= 222:00 +CPU= 56.581
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000888
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000888
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000888
  14:01:23 STEP  889 H= 222:15 +CPU= 45.580
  14:02:09 STEP  890 H= 222:30 +CPU= 45.272
  14:02:55 STEP  891 H= 222:45 +CPU= 45.330
  14:03:42 STEP  892 H= 223:00 +CPU= 46.499
  14:04:27 STEP  893 H= 223:15 +CPU= 44.517
  14:05:11 STEP  894 H= 223:30 +CPU= 44.397
  14:05:57 STEP  895 H= 223:45 +CPU= 45.109
  14:06:44 STEP  896 H= 224:00 +CPU= 46.252
  14:07:30 STEP  897 H= 224:15 +CPU= 45.017
  14:08:15 STEP  898 H= 224:30 +CPU= 44.952
  14:09:01 STEP  899 H= 224:45 +CPU= 44.639
  14:09:58 STEP  900 H= 225:00 +CPU= 56.423
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000900
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000900
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000900
  14:10:43 STEP  901 H= 225:15 +CPU= 44.253
  14:11:29 STEP  902 H= 225:30 +CPU= 45.762
  14:12:15 STEP  903 H= 225:45 +CPU= 46.187
  14:13:02 STEP  904 H= 226:00 +CPU= 46.239
  14:13:48 STEP  905 H= 226:15 +CPU= 45.256
  14:14:33 STEP  906 H= 226:30 +CPU= 45.425
  14:15:18 STEP  907 H= 226:45 +CPU= 44.690
  14:16:05 STEP  908 H= 227:00 +CPU= 45.949
  14:16:51 STEP  909 H= 227:15 +CPU= 45.751
  14:17:37 STEP  910 H= 227:30 +CPU= 45.497
  14:18:23 STEP  911 H= 227:45 +CPU= 45.516
  14:19:23 STEP  912 H= 228:00 +CPU= 59.846
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000912
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000912
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000912
  14:20:10 STEP  913 H= 228:15 +CPU= 45.964
  14:20:55 STEP  914 H= 228:30 +CPU= 45.079
  14:21:41 STEP  915 H= 228:45 +CPU= 45.333
  14:22:27 STEP  916 H= 229:00 +CPU= 46.044
  14:23:13 STEP  917 H= 229:15 +CPU= 44.962
  14:24:01 STEP  918 H= 229:30 +CPU= 45.033
  14:24:49 STEP  919 H= 229:45 +CPU= 45.458
  14:25:39 STEP  920 H= 230:00 +CPU= 46.054
  14:26:24 STEP  921 H= 230:15 +CPU= 44.884
  14:27:09 STEP  922 H= 230:30 +CPU= 44.282
  14:27:54 STEP  923 H= 230:45 +CPU= 44.612
  14:28:49 STEP  924 H= 231:00 +CPU= 54.817
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000924
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000924
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000924
  14:29:35 STEP  925 H= 231:15 +CPU= 45.055
  14:30:21 STEP  926 H= 231:30 +CPU= 45.040
  14:31:06 STEP  927 H= 231:45 +CPU= 44.932
  14:31:52 STEP  928 H= 232:00 +CPU= 45.841
  14:32:37 STEP  929 H= 232:15 +CPU= 44.297
  14:33:23 STEP  930 H= 232:30 +CPU= 45.106
  14:34:08 STEP  931 H= 232:45 +CPU= 45.150
  14:34:55 STEP  932 H= 233:00 +CPU= 45.953
  14:35:41 STEP  933 H= 233:15 +CPU= 45.720
  14:36:27 STEP  934 H= 233:30 +CPU= 46.034
  14:37:13 STEP  935 H= 233:45 +CPU= 45.567
  14:38:09 STEP  936 H= 234:00 +CPU= 55.372
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000936
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000936
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000936
  14:38:55 STEP  937 H= 234:15 +CPU= 45.488
  14:39:41 STEP  938 H= 234:30 +CPU= 45.306
  14:40:28 STEP  939 H= 234:45 +CPU= 44.934
  14:41:14 STEP  940 H= 235:00 +CPU= 45.268
  14:42:01 STEP  941 H= 235:15 +CPU= 45.168
  14:42:46 STEP  942 H= 235:30 +CPU= 44.981
  14:43:31 STEP  943 H= 235:45 +CPU= 44.282
  14:44:16 STEP  944 H= 236:00 +CPU= 45.234
  14:45:02 STEP  945 H= 236:15 +CPU= 44.830
  14:45:47 STEP  946 H= 236:30 +CPU= 45.283
  14:46:33 STEP  947 H= 236:45 +CPU= 45.619
  14:47:30 STEP  948 H= 237:00 +CPU= 55.570
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000948
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000948
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000948
  14:48:15 STEP  949 H= 237:15 +CPU= 44.956
  14:49:00 STEP  950 H= 237:30 +CPU= 44.937
  14:49:46 STEP  951 H= 237:45 +CPU= 45.493
  14:50:33 STEP  952 H= 238:00 +CPU= 46.122
  14:51:18 STEP  953 H= 238:15 +CPU= 44.855
  14:52:04 STEP  954 H= 238:30 +CPU= 45.079
  14:52:49 STEP  955 H= 238:45 +CPU= 44.601
  14:53:35 STEP  956 H= 239:00 +CPU= 45.715
  14:54:21 STEP  957 H= 239:15 +CPU= 46.053
  14:55:08 STEP  958 H= 239:30 +CPU= 46.736
  14:55:54 STEP  959 H= 239:45 +CPU= 45.340
  14:56:54 STEP  960 H= 240:00 +CPU= 58.939
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000960
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000960
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  14:57:40 STEP  961 H= 240:15 +CPU= 44.546
  14:58:25 STEP  962 H= 240:30 +CPU= 45.162
  14:59:11 STEP  963 H= 240:45 +CPU= 45.580
  14:59:57 STEP  964 H= 241:00 +CPU= 45.233
  15:00:43 STEP  965 H= 241:15 +CPU= 45.870
  15:01:29 STEP  966 H= 241:30 +CPU= 45.520
  15:02:14 STEP  967 H= 241:45 +CPU= 44.502
  15:03:01 STEP  968 H= 242:00 +CPU= 46.819
  15:03:47 STEP  969 H= 242:15 +CPU= 45.034
  15:04:33 STEP  970 H= 242:30 +CPU= 46.277
  15:05:19 STEP  971 H= 242:45 +CPU= 45.162
  15:06:15 STEP  972 H= 243:00 +CPU= 56.090
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000972
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000972
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000972
  15:07:01 STEP  973 H= 243:15 +CPU= 45.123
  15:07:46 STEP  974 H= 243:30 +CPU= 44.742
  15:08:31 STEP  975 H= 243:45 +CPU= 44.924
  15:09:18 STEP  976 H= 244:00 +CPU= 45.750
  15:10:04 STEP  977 H= 244:15 +CPU= 45.750
  15:10:51 STEP  978 H= 244:30 +CPU= 46.555
  15:11:36 STEP  979 H= 244:45 +CPU= 45.400
  15:12:24 STEP  980 H= 245:00 +CPU= 46.564
  15:13:11 STEP  981 H= 245:15 +CPU= 46.033
  15:13:57 STEP  982 H= 245:30 +CPU= 45.778
  15:14:43 STEP  983 H= 245:45 +CPU= 45.806
  15:15:38 STEP  984 H= 246:00 +CPU= 54.433
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000984
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000984
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000984
  15:16:24 STEP  985 H= 246:15 +CPU= 45.585
  15:17:10 STEP  986 H= 246:30 +CPU= 45.267
  15:17:56 STEP  987 H= 246:45 +CPU= 45.481
  15:18:42 STEP  988 H= 247:00 +CPU= 46.267
  15:19:28 STEP  989 H= 247:15 +CPU= 45.672
  15:20:13 STEP  990 H= 247:30 +CPU= 44.509
  15:20:58 STEP  991 H= 247:45 +CPU= 44.691
  15:21:44 STEP  992 H= 248:00 +CPU= 44.809
  15:22:29 STEP  993 H= 248:15 +CPU= 45.497
  15:23:15 STEP  994 H= 248:30 +CPU= 45.240
  15:24:01 STEP  995 H= 248:45 +CPU= 45.273
  15:24:57 STEP  996 H= 249:00 +CPU= 54.646
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+000996
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+000996
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+000996
  15:25:43 STEP  997 H= 249:15 +CPU= 45.071
  15:26:28 STEP  998 H= 249:30 +CPU= 44.788
  15:27:13 STEP  999 H= 249:45 +CPU= 44.009
  15:27:59 STEP 1000 H= 250:00 +CPU= 45.466
  15:28:44 STEP 1001 H= 250:15 +CPU= 45.271
  15:29:30 STEP 1002 H= 250:30 +CPU= 45.440
  15:30:16 STEP 1003 H= 250:45 +CPU= 45.891
  15:31:02 STEP 1004 H= 251:00 +CPU= 45.209
  15:31:48 STEP 1005 H= 251:15 +CPU= 45.074
  15:32:33 STEP 1006 H= 251:30 +CPU= 44.538
  15:33:18 STEP 1007 H= 251:45 +CPU= 45.292
  15:34:19 STEP 1008 H= 252:00 +CPU= 60.309
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001008
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001008
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001008
  15:35:05 STEP 1009 H= 252:15 +CPU= 45.319
  15:35:51 STEP 1010 H= 252:30 +CPU= 45.397
  15:36:36 STEP 1011 H= 252:45 +CPU= 44.652
  15:37:22 STEP 1012 H= 253:00 +CPU= 45.439
  15:38:07 STEP 1013 H= 253:15 +CPU= 44.953
  15:38:53 STEP 1014 H= 253:30 +CPU= 45.319
  15:39:39 STEP 1015 H= 253:45 +CPU= 45.853
  15:40:24 STEP 1016 H= 254:00 +CPU= 44.676
  15:41:11 STEP 1017 H= 254:15 +CPU= 46.186
  15:41:56 STEP 1018 H= 254:30 +CPU= 45.134
  15:42:42 STEP 1019 H= 254:45 +CPU= 44.960
  15:43:38 STEP 1020 H= 255:00 +CPU= 55.563
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001020
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001020
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001020
  15:44:24 STEP 1021 H= 255:15 +CPU= 46.463
  15:45:10 STEP 1022 H= 255:30 +CPU= 45.532
  15:45:56 STEP 1023 H= 255:45 +CPU= 44.669
  15:46:42 STEP 1024 H= 256:00 +CPU= 45.830
  15:47:27 STEP 1025 H= 256:15 +CPU= 45.245
  15:48:13 STEP 1026 H= 256:30 +CPU= 45.044
  15:48:59 STEP 1027 H= 256:45 +CPU= 45.699
  15:49:45 STEP 1028 H= 257:00 +CPU= 45.671
  15:50:31 STEP 1029 H= 257:15 +CPU= 44.906
  15:51:15 STEP 1030 H= 257:30 +CPU= 44.247
  15:52:01 STEP 1031 H= 257:45 +CPU= 45.532
  15:52:57 STEP 1032 H= 258:00 +CPU= 55.207
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001032
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001032
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001032
  15:53:43 STEP 1033 H= 258:15 +CPU= 45.580
  15:54:28 STEP 1034 H= 258:30 +CPU= 44.803
  15:55:13 STEP 1035 H= 258:45 +CPU= 44.527
  15:55:59 STEP 1036 H= 259:00 +CPU= 45.380
  15:56:44 STEP 1037 H= 259:15 +CPU= 44.970
  15:57:29 STEP 1038 H= 259:30 +CPU= 44.456
  15:58:15 STEP 1039 H= 259:45 +CPU= 45.321
  15:59:00 STEP 1040 H= 260:00 +CPU= 44.762
  15:59:45 STEP 1041 H= 260:15 +CPU= 44.706
  16:00:31 STEP 1042 H= 260:30 +CPU= 45.006
  16:01:16 STEP 1043 H= 260:45 +CPU= 44.907
  16:02:12 STEP 1044 H= 261:00 +CPU= 55.064
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001044
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001044
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001044
  16:02:57 STEP 1045 H= 261:15 +CPU= 44.803
  16:03:42 STEP 1046 H= 261:30 +CPU= 45.478
  16:04:28 STEP 1047 H= 261:45 +CPU= 45.169
  16:05:14 STEP 1048 H= 262:00 +CPU= 45.315
  16:05:59 STEP 1049 H= 262:15 +CPU= 44.643
  16:06:44 STEP 1050 H= 262:30 +CPU= 45.269
  16:07:30 STEP 1051 H= 262:45 +CPU= 45.130
  16:08:17 STEP 1052 H= 263:00 +CPU= 46.284
  16:09:02 STEP 1053 H= 263:15 +CPU= 45.576
  16:09:49 STEP 1054 H= 263:30 +CPU= 46.184
  16:10:35 STEP 1055 H= 263:45 +CPU= 45.457
  16:11:35 STEP 1056 H= 264:00 +CPU= 59.477
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001056
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001056
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_10.zip
Uploading the intermediate file: upload_file_10.zip
  16:12:21 STEP 1057 H= 264:15 +CPU= 44.300
  16:13:07 STEP 1058 H= 264:30 +CPU= 45.210
  16:13:52 STEP 1059 H= 264:45 +CPU= 45.070
  16:14:38 STEP 1060 H= 265:00 +CPU= 45.100
  16:15:24 STEP 1061 H= 265:15 +CPU= 46.235
  16:16:10 STEP 1062 H= 265:30 +CPU= 45.335
  16:16:56 STEP 1063 H= 265:45 +CPU= 45.569
  16:17:42 STEP 1064 H= 266:00 +CPU= 45.489
  16:18:27 STEP 1065 H= 266:15 +CPU= 44.865
  16:19:13 STEP 1066 H= 266:30 +CPU= 45.724
  16:19:59 STEP 1067 H= 266:45 +CPU= 45.174
  16:20:55 STEP 1068 H= 267:00 +CPU= 55.302
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001068
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001068
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001068
  16:21:40 STEP 1069 H= 267:15 +CPU= 45.050
  16:22:25 STEP 1070 H= 267:30 +CPU= 44.241
  16:23:10 STEP 1071 H= 267:45 +CPU= 45.066
  16:23:56 STEP 1072 H= 268:00 +CPU= 45.398
  16:24:42 STEP 1073 H= 268:15 +CPU= 46.201
  16:25:27 STEP 1074 H= 268:30 +CPU= 44.716
  16:26:12 STEP 1075 H= 268:45 +CPU= 44.268
  16:26:59 STEP 1076 H= 269:00 +CPU= 46.869
  16:27:45 STEP 1077 H= 269:15 +CPU= 44.865
  16:28:32 STEP 1078 H= 269:30 +CPU= 46.232
  16:29:18 STEP 1079 H= 269:45 +CPU= 45.262
  16:30:13 STEP 1080 H= 270:00 +CPU= 55.291
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001080
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001080
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001080
  16:31:00 STEP 1081 H= 270:15 +CPU= 45.732
  16:31:45 STEP 1082 H= 270:30 +CPU= 45.188
  16:32:31 STEP 1083 H= 270:45 +CPU= 45.798
  16:33:18 STEP 1084 H= 271:00 +CPU= 46.503
  16:34:03 STEP 1085 H= 271:15 +CPU= 44.681
  16:34:48 STEP 1086 H= 271:30 +CPU= 44.532
  16:35:33 STEP 1087 H= 271:45 +CPU= 44.907
  16:36:19 STEP 1088 H= 272:00 +CPU= 45.446
  16:37:05 STEP 1089 H= 272:15 +CPU= 45.368
  16:37:51 STEP 1090 H= 272:30 +CPU= 45.758
  16:38:37 STEP 1091 H= 272:45 +CPU= 45.792
  16:39:34 STEP 1092 H= 273:00 +CPU= 55.552
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001092
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001092
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001092
  16:40:19 STEP 1093 H= 273:15 +CPU= 44.689
  16:41:05 STEP 1094 H= 273:30 +CPU= 45.826
  16:41:51 STEP 1095 H= 273:45 +CPU= 45.853
  16:42:39 STEP 1096 H= 274:00 +CPU= 47.442
  16:43:26 STEP 1097 H= 274:15 +CPU= 45.581
  16:44:13 STEP 1098 H= 274:30 +CPU= 45.968
  16:44:58 STEP 1099 H= 274:45 +CPU= 45.146
  16:45:44 STEP 1100 H= 275:00 +CPU= 45.040
  16:46:31 STEP 1101 H= 275:15 +CPU= 45.900
  16:47:17 STEP 1102 H= 275:30 +CPU= 45.885
  16:48:03 STEP 1103 H= 275:45 +CPU= 45.100
  16:49:04 STEP 1104 H= 276:00 +CPU= 60.992
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001104
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001104
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001104
  16:49:50 STEP 1105 H= 276:15 +CPU= 45.314
  16:50:36 STEP 1106 H= 276:30 +CPU= 45.950
  16:51:23 STEP 1107 H= 276:45 +CPU= 46.137
  16:52:09 STEP 1108 H= 277:00 +CPU= 45.369
  16:52:54 STEP 1109 H= 277:15 +CPU= 44.898
  16:53:39 STEP 1110 H= 277:30 +CPU= 45.032
  16:54:26 STEP 1111 H= 277:45 +CPU= 45.784
  16:55:14 STEP 1112 H= 278:00 +CPU= 47.731
  16:55:59 STEP 1113 H= 278:15 +CPU= 45.147
  16:56:45 STEP 1114 H= 278:30 +CPU= 45.736
  16:57:31 STEP 1115 H= 278:45 +CPU= 45.622
  16:58:28 STEP 1116 H= 279:00 +CPU= 56.459
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001116
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001116
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001116
  16:59:14 STEP 1117 H= 279:15 +CPU= 45.555
  16:59:59 STEP 1118 H= 279:30 +CPU= 43.845
  17:00:45 STEP 1119 H= 279:45 +CPU= 45.549
  17:01:31 STEP 1120 H= 280:00 +CPU= 45.574
  17:02:17 STEP 1121 H= 280:15 +CPU= 45.702
  17:03:02 STEP 1122 H= 280:30 +CPU= 44.717
  17:03:47 STEP 1123 H= 280:45 +CPU= 45.316
  17:04:34 STEP 1124 H= 281:00 +CPU= 45.917
  17:05:19 STEP 1125 H= 281:15 +CPU= 45.020
  17:06:04 STEP 1126 H= 281:30 +CPU= 44.867
  17:06:50 STEP 1127 H= 281:45 +CPU= 45.342
  17:07:46 STEP 1128 H= 282:00 +CPU= 55.744
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001128
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001128
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001128
  17:08:33 STEP 1129 H= 282:15 +CPU= 45.916
  17:09:18 STEP 1130 H= 282:30 +CPU= 44.808
  17:10:03 STEP 1131 H= 282:45 +CPU= 45.294
  17:10:50 STEP 1132 H= 283:00 +CPU= 46.618
  17:11:35 STEP 1133 H= 283:15 +CPU= 44.307
  17:12:20 STEP 1134 H= 283:30 +CPU= 43.967
  17:13:04 STEP 1135 H= 283:45 +CPU= 43.745
  17:13:50 STEP 1136 H= 284:00 +CPU= 45.695
  17:14:36 STEP 1137 H= 284:15 +CPU= 45.588
  17:15:22 STEP 1138 H= 284:30 +CPU= 45.554
  17:16:08 STEP 1139 H= 284:45 +CPU= 46.330
  17:17:06 STEP 1140 H= 285:00 +CPU= 56.800
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001140
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001140
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001140
  17:17:52 STEP 1141 H= 285:15 +CPU= 46.303
  17:18:37 STEP 1142 H= 285:30 +CPU= 44.698
  17:19:23 STEP 1143 H= 285:45 +CPU= 45.078
  17:20:10 STEP 1144 H= 286:00 +CPU= 47.305
  17:20:56 STEP 1145 H= 286:15 +CPU= 45.206
  17:21:43 STEP 1146 H= 286:30 +CPU= 46.583
  17:22:29 STEP 1147 H= 286:45 +CPU= 45.922
  17:23:15 STEP 1148 H= 287:00 +CPU= 45.520
  17:24:01 STEP 1149 H= 287:15 +CPU= 45.060
  17:24:48 STEP 1150 H= 287:30 +CPU= 46.402
  17:25:33 STEP 1151 H= 287:45 +CPU= 45.204
  17:26:34 STEP 1152 H= 288:00 +CPU= 60.424
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001152
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001152
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_11.zip
Uploading the intermediate file: upload_file_11.zip
  17:27:20 STEP 1153 H= 288:15 +CPU= 44.629
  17:28:06 STEP 1154 H= 288:30 +CPU= 44.840
  17:28:51 STEP 1155 H= 288:45 +CPU= 45.084
  17:29:37 STEP 1156 H= 289:00 +CPU= 45.440
  17:30:23 STEP 1157 H= 289:15 +CPU= 45.992
  17:31:10 STEP 1158 H= 289:30 +CPU= 46.281
  17:31:55 STEP 1159 H= 289:45 +CPU= 44.468
  17:32:42 STEP 1160 H= 290:00 +CPU= 46.474
  17:33:27 STEP 1161 H= 290:15 +CPU= 45.042
  17:34:13 STEP 1162 H= 290:30 +CPU= 45.725
  17:34:59 STEP 1163 H= 290:45 +CPU= 45.007
  17:35:55 STEP 1164 H= 291:00 +CPU= 55.587
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001164
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001164
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001164
  17:36:40 STEP 1165 H= 291:15 +CPU= 45.315
  17:37:26 STEP 1166 H= 291:30 +CPU= 44.914
  17:38:11 STEP 1167 H= 291:45 +CPU= 44.961
  17:38:57 STEP 1168 H= 292:00 +CPU= 45.251
  17:39:43 STEP 1169 H= 292:15 +CPU= 45.329
  17:40:29 STEP 1170 H= 292:30 +CPU= 45.983
  17:41:14 STEP 1171 H= 292:45 +CPU= 44.487
  17:42:00 STEP 1172 H= 293:00 +CPU= 45.994
  17:42:46 STEP 1173 H= 293:15 +CPU= 44.752
  17:43:33 STEP 1174 H= 293:30 +CPU= 45.896
  17:44:18 STEP 1175 H= 293:45 +CPU= 45.254
  17:45:15 STEP 1176 H= 294:00 +CPU= 56.217
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001176
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001176
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001176
  17:46:01 STEP 1177 H= 294:15 +CPU= 45.620
  17:46:47 STEP 1178 H= 294:30 +CPU= 45.332
  17:47:32 STEP 1179 H= 294:45 +CPU= 44.945
  17:48:19 STEP 1180 H= 295:00 +CPU= 46.430
  17:49:03 STEP 1181 H= 295:15 +CPU= 44.157
  17:49:49 STEP 1182 H= 295:30 +CPU= 45.498
  17:50:34 STEP 1183 H= 295:45 +CPU= 44.528
  17:51:20 STEP 1184 H= 296:00 +CPU= 45.422
  17:52:05 STEP 1185 H= 296:15 +CPU= 45.005
  17:52:52 STEP 1186 H= 296:30 +CPU= 46.280
  17:53:38 STEP 1187 H= 296:45 +CPU= 45.611
  17:54:35 STEP 1188 H= 297:00 +CPU= 56.978
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001188
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001188
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001188
  17:55:21 STEP 1189 H= 297:15 +CPU= 45.073
  17:56:06 STEP 1190 H= 297:30 +CPU= 45.276
  17:56:52 STEP 1191 H= 297:45 +CPU= 44.714
  17:57:38 STEP 1192 H= 298:00 +CPU= 45.673
  17:58:22 STEP 1193 H= 298:15 +CPU= 44.448
  17:59:08 STEP 1194 H= 298:30 +CPU= 45.463
  17:59:54 STEP 1195 H= 298:45 +CPU= 44.933
  18:00:41 STEP 1196 H= 299:00 +CPU= 46.528
  18:01:26 STEP 1197 H= 299:15 +CPU= 44.120
  18:02:11 STEP 1198 H= 299:30 +CPU= 44.639
  18:02:58 STEP 1199 H= 299:45 +CPU= 46.658
  18:03:59 STEP 1200 H= 300:00 +CPU= 59.700
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001200
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001200
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001200
  18:04:44 STEP 1201 H= 300:15 +CPU= 44.547
  18:05:30 STEP 1202 H= 300:30 +CPU= 45.588
  18:06:15 STEP 1203 H= 300:45 +CPU= 45.099
  18:07:02 STEP 1204 H= 301:00 +CPU= 46.236
  18:07:48 STEP 1205 H= 301:15 +CPU= 45.435
  18:08:32 STEP 1206 H= 301:30 +CPU= 44.047
  18:09:18 STEP 1207 H= 301:45 +CPU= 45.294
  18:10:04 STEP 1208 H= 302:00 +CPU= 46.053
  18:10:51 STEP 1209 H= 302:15 +CPU= 45.804
  18:11:36 STEP 1210 H= 302:30 +CPU= 44.842
  18:12:22 STEP 1211 H= 302:45 +CPU= 45.515
  18:13:18 STEP 1212 H= 303:00 +CPU= 55.631
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001212
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001212
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001212
  18:14:03 STEP 1213 H= 303:15 +CPU= 44.829
  18:14:48 STEP 1214 H= 303:30 +CPU= 44.387
  18:15:33 STEP 1215 H= 303:45 +CPU= 44.944
  18:16:20 STEP 1216 H= 304:00 +CPU= 46.177
  18:17:05 STEP 1217 H= 304:15 +CPU= 45.324
  18:17:50 STEP 1218 H= 304:30 +CPU= 44.281
  18:18:36 STEP 1219 H= 304:45 +CPU= 45.512
  18:19:22 STEP 1220 H= 305:00 +CPU= 45.172
  18:20:08 STEP 1221 H= 305:15 +CPU= 46.237
  18:20:53 STEP 1222 H= 305:30 +CPU= 44.627
  18:21:39 STEP 1223 H= 305:45 +CPU= 45.080
  18:22:36 STEP 1224 H= 306:00 +CPU= 56.728
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001224
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001224
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001224
  18:23:21 STEP 1225 H= 306:15 +CPU= 44.793
  18:24:07 STEP 1226 H= 306:30 +CPU= 45.624
  18:24:53 STEP 1227 H= 306:45 +CPU= 45.172
  18:25:39 STEP 1228 H= 307:00 +CPU= 45.809
  18:26:25 STEP 1229 H= 307:15 +CPU= 45.720
  18:27:09 STEP 1230 H= 307:30 +CPU= 43.992
  18:27:55 STEP 1231 H= 307:45 +CPU= 44.713
  18:28:41 STEP 1232 H= 308:00 +CPU= 46.385
  18:29:26 STEP 1233 H= 308:15 +CPU= 44.506
  18:30:12 STEP 1234 H= 308:30 +CPU= 45.606
  18:30:56 STEP 1235 H= 308:45 +CPU= 43.646
  18:31:53 STEP 1236 H= 309:00 +CPU= 56.781
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001236
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001236
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001236
  18:32:40 STEP 1237 H= 309:15 +CPU= 46.240
  18:33:25 STEP 1238 H= 309:30 +CPU= 44.606
  18:34:11 STEP 1239 H= 309:45 +CPU= 45.704
  18:34:57 STEP 1240 H= 310:00 +CPU= 45.046
  18:35:41 STEP 1241 H= 310:15 +CPU= 44.424
  18:36:26 STEP 1242 H= 310:30 +CPU= 44.671
  18:37:11 STEP 1243 H= 310:45 +CPU= 44.668
  18:37:57 STEP 1244 H= 311:00 +CPU= 45.548
  18:38:43 STEP 1245 H= 311:15 +CPU= 45.313
  18:39:30 STEP 1246 H= 311:30 +CPU= 46.144
  18:40:14 STEP 1247 H= 311:45 +CPU= 44.187
  18:41:17 STEP 1248 H= 312:00 +CPU= 61.738
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001248
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001248
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_12.zip
Uploading the intermediate file: upload_file_12.zip
  18:42:01 STEP 1249 H= 312:15 +CPU= 43.011
  18:42:46 STEP 1250 H= 312:30 +CPU= 44.462
  18:43:32 STEP 1251 H= 312:45 +CPU= 44.882
  18:44:17 STEP 1252 H= 313:00 +CPU= 45.479
  18:45:02 STEP 1253 H= 313:15 +CPU= 44.199
  18:45:48 STEP 1254 H= 313:30 +CPU= 45.249
  18:46:33 STEP 1255 H= 313:45 +CPU= 44.884
  18:47:20 STEP 1256 H= 314:00 +CPU= 47.007
  18:48:07 STEP 1257 H= 314:15 +CPU= 46.532
  18:48:55 STEP 1258 H= 314:30 +CPU= 47.181
  18:49:44 STEP 1259 H= 314:45 +CPU= 49.165
  18:50:43 STEP 1260 H= 315:00 +CPU= 58.381
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001260
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001260
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001260
  18:51:32 STEP 1261 H= 315:15 +CPU= 48.624
  18:52:20 STEP 1262 H= 315:30 +CPU= 47.105
  18:53:11 STEP 1263 H= 315:45 +CPU= 50.606
  18:54:01 STEP 1264 H= 316:00 +CPU= 49.671
  18:54:49 STEP 1265 H= 316:15 +CPU= 47.684
  18:55:38 STEP 1266 H= 316:30 +CPU= 48.705
  18:56:27 STEP 1267 H= 316:45 +CPU= 48.853
  18:57:18 STEP 1268 H= 317:00 +CPU= 49.940
  18:58:03 STEP 1269 H= 317:15 +CPU= 45.361
  18:58:47 STEP 1270 H= 317:30 +CPU= 43.689
  18:59:32 STEP 1271 H= 317:45 +CPU= 44.279
  19:00:27 STEP 1272 H= 318:00 +CPU= 54.761
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001272
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001272
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001272
  19:01:11 STEP 1273 H= 318:15 +CPU= 43.526
  19:01:56 STEP 1274 H= 318:30 +CPU= 44.578
  19:02:41 STEP 1275 H= 318:45 +CPU= 44.225
  19:03:29 STEP 1276 H= 319:00 +CPU= 47.925
  19:04:16 STEP 1277 H= 319:15 +CPU= 46.295
  19:05:05 STEP 1278 H= 319:30 +CPU= 48.725
  19:05:52 STEP 1279 H= 319:45 +CPU= 47.043
  19:06:40 STEP 1280 H= 320:00 +CPU= 47.522
  19:07:28 STEP 1281 H= 320:15 +CPU= 47.635
  19:08:15 STEP 1282 H= 320:30 +CPU= 46.017
  19:09:02 STEP 1283 H= 320:45 +CPU= 46.936
  19:10:00 STEP 1284 H= 321:00 +CPU= 57.734
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001284
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001284
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001284
  19:10:46 STEP 1285 H= 321:15 +CPU= 45.676
  19:11:34 STEP 1286 H= 321:30 +CPU= 47.295
  19:12:22 STEP 1287 H= 321:45 +CPU= 47.180
  19:13:09 STEP 1288 H= 322:00 +CPU= 46.396
  19:13:58 STEP 1289 H= 322:15 +CPU= 48.608
  19:14:44 STEP 1290 H= 322:30 +CPU= 46.457
  19:15:30 STEP 1291 H= 322:45 +CPU= 44.748
  19:16:16 STEP 1292 H= 323:00 +CPU= 45.516
  19:17:01 STEP 1293 H= 323:15 +CPU= 45.151
  19:17:47 STEP 1294 H= 323:30 +CPU= 45.853
  19:18:35 STEP 1295 H= 323:45 +CPU= 47.180
  19:19:40 STEP 1296 H= 324:00 +CPU= 64.185
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001296
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001296
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001296
  19:20:27 STEP 1297 H= 324:15 +CPU= 46.840
  19:21:11 STEP 1298 H= 324:30 +CPU= 44.042
  19:21:54 STEP 1299 H= 324:45 +CPU= 42.344
  19:22:33 STEP 1300 H= 325:00 +CPU= 39.206
  19:23:12 STEP 1301 H= 325:15 +CPU= 38.461
  19:23:50 STEP 1302 H= 325:30 +CPU= 37.928
  19:24:29 STEP 1303 H= 325:45 +CPU= 37.705
  19:25:11 STEP 1304 H= 326:00 +CPU= 41.986
  19:25:51 STEP 1305 H= 326:15 +CPU= 39.555
  19:26:31 STEP 1306 H= 326:30 +CPU= 40.275
  19:27:12 STEP 1307 H= 326:45 +CPU= 40.516
  19:28:05 STEP 1308 H= 327:00 +CPU= 52.172
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001308
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001308
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001308
  19:28:48 STEP 1309 H= 327:15 +CPU= 43.225
  19:29:31 STEP 1310 H= 327:30 +CPU= 42.586
  19:30:14 STEP 1311 H= 327:45 +CPU= 42.491
  19:30:58 STEP 1312 H= 328:00 +CPU= 43.337
  19:31:41 STEP 1313 H= 328:15 +CPU= 42.907
  19:32:24 STEP 1314 H= 328:30 +CPU= 42.805
  19:33:08 STEP 1315 H= 328:45 +CPU= 43.378
  19:33:51 STEP 1316 H= 329:00 +CPU= 42.650
  19:34:34 STEP 1317 H= 329:15 +CPU= 42.661
  19:35:16 STEP 1318 H= 329:30 +CPU= 41.569
  19:36:00 STEP 1319 H= 329:45 +CPU= 43.712
  19:36:56 STEP 1320 H= 330:00 +CPU= 55.335
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001320
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001320
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001320
  19:37:39 STEP 1321 H= 330:15 +CPU= 42.760
  19:38:23 STEP 1322 H= 330:30 +CPU= 43.914
  19:39:10 STEP 1323 H= 330:45 +CPU= 45.898
  19:39:59 STEP 1324 H= 331:00 +CPU= 48.485
  19:40:47 STEP 1325 H= 331:15 +CPU= 47.451
  19:41:35 STEP 1326 H= 331:30 +CPU= 47.670
  19:42:21 STEP 1327 H= 331:45 +CPU= 46.282
  19:43:10 STEP 1328 H= 332:00 +CPU= 48.150
  19:43:57 STEP 1329 H= 332:15 +CPU= 46.941
  19:44:44 STEP 1330 H= 332:30 +CPU= 46.784
  19:45:35 STEP 1331 H= 332:45 +CPU= 50.446
  19:46:33 STEP 1332 H= 333:00 +CPU= 57.247
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001332
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001332
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001332
  19:47:22 STEP 1333 H= 333:15 +CPU= 48.114
  19:48:09 STEP 1334 H= 333:30 +CPU= 46.882
  19:49:01 STEP 1335 H= 333:45 +CPU= 51.318
  19:49:47 STEP 1336 H= 334:00 +CPU= 46.202
  19:50:32 STEP 1337 H= 334:15 +CPU= 44.112
  19:51:15 STEP 1338 H= 334:30 +CPU= 43.010
  19:52:01 STEP 1339 H= 334:45 +CPU= 45.665
  19:52:47 STEP 1340 H= 335:00 +CPU= 45.756
  19:53:35 STEP 1341 H= 335:15 +CPU= 46.793
  19:54:22 STEP 1342 H= 335:30 +CPU= 47.073
  19:55:10 STEP 1343 H= 335:45 +CPU= 47.519
  19:56:16 STEP 1344 H= 336:00 +CPU= 65.056
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001344
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001344
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_13.zip
Uploading the intermediate file: upload_file_13.zip
  19:57:03 STEP 1345 H= 336:15 +CPU= 46.047
  19:57:50 STEP 1346 H= 336:30 +CPU= 45.752
  19:58:39 STEP 1347 H= 336:45 +CPU= 48.503
  19:59:30 STEP 1348 H= 337:00 +CPU= 50.544
  20:00:19 STEP 1349 H= 337:15 +CPU= 48.334
  20:01:08 STEP 1350 H= 337:30 +CPU= 49.035
  20:01:57 STEP 1351 H= 337:45 +CPU= 48.193
  20:02:44 STEP 1352 H= 338:00 +CPU= 46.240
  20:03:29 STEP 1353 H= 338:15 +CPU= 44.727
  20:04:13 STEP 1354 H= 338:30 +CPU= 43.844
  20:04:55 STEP 1355 H= 338:45 +CPU= 41.718
  20:05:51 STEP 1356 H= 339:00 +CPU= 54.906
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001356
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001356
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001356
  20:06:36 STEP 1357 H= 339:15 +CPU= 45.027
  20:07:23 STEP 1358 H= 339:30 +CPU= 46.883
  20:08:13 STEP 1359 H= 339:45 +CPU= 49.407
  20:09:01 STEP 1360 H= 340:00 +CPU= 47.597
  20:09:54 STEP 1361 H= 340:15 +CPU= 48.764
  20:10:41 STEP 1362 H= 340:30 +CPU= 45.770
  20:11:30 STEP 1363 H= 340:45 +CPU= 47.758
  20:12:17 STEP 1364 H= 341:00 +CPU= 46.375
  20:13:08 STEP 1365 H= 341:15 +CPU= 50.352
  20:13:55 STEP 1366 H= 341:30 +CPU= 46.350
  20:14:46 STEP 1367 H= 341:45 +CPU= 49.544
  20:15:45 STEP 1368 H= 342:00 +CPU= 57.738
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001368
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001368
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001368
  20:16:33 STEP 1369 H= 342:15 +CPU= 47.367
  20:17:19 STEP 1370 H= 342:30 +CPU= 44.781
  20:18:03 STEP 1371 H= 342:45 +CPU= 42.880
  20:18:50 STEP 1372 H= 343:00 +CPU= 45.644
  20:19:37 STEP 1373 H= 343:15 +CPU= 45.979
  20:20:25 STEP 1374 H= 343:30 +CPU= 46.164
  20:21:14 STEP 1375 H= 343:45 +CPU= 48.321
  20:22:05 STEP 1376 H= 344:00 +CPU= 49.368
  20:22:55 STEP 1377 H= 344:15 +CPU= 47.790
  20:23:46 STEP 1378 H= 344:30 +CPU= 48.015
  20:24:37 STEP 1379 H= 344:45 +CPU= 48.446
  20:25:37 STEP 1380 H= 345:00 +CPU= 58.563
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001380
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001380
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001380
  20:26:26 STEP 1381 H= 345:15 +CPU= 48.029
  20:27:16 STEP 1382 H= 345:30 +CPU= 48.493
  20:28:05 STEP 1383 H= 345:45 +CPU= 47.251
  20:28:53 STEP 1384 H= 346:00 +CPU= 47.266
  20:29:43 STEP 1385 H= 346:15 +CPU= 47.626
  20:30:30 STEP 1386 H= 346:30 +CPU= 45.345
  20:31:17 STEP 1387 H= 346:45 +CPU= 45.545
  20:32:02 STEP 1388 H= 347:00 +CPU= 43.975
  20:32:50 STEP 1389 H= 347:15 +CPU= 46.025
  20:33:37 STEP 1390 H= 347:30 +CPU= 44.885
  20:34:26 STEP 1391 H= 347:45 +CPU= 47.947
  20:35:34 STEP 1392 H= 348:00 +CPU= 63.102
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001392
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001392
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001392
  20:36:24 STEP 1393 H= 348:15 +CPU= 46.673
  20:37:15 STEP 1394 H= 348:30 +CPU= 47.538
  20:38:09 STEP 1395 H= 348:45 +CPU= 48.756
  20:39:02 STEP 1396 H= 349:00 +CPU= 48.341
  20:39:53 STEP 1397 H= 349:15 +CPU= 48.660
  20:40:41 STEP 1398 H= 349:30 +CPU= 47.620
  20:41:28 STEP 1399 H= 349:45 +CPU= 46.612
  20:42:17 STEP 1400 H= 350:00 +CPU= 48.221
  20:43:05 STEP 1401 H= 350:15 +CPU= 48.141
  20:43:52 STEP 1402 H= 350:30 +CPU= 46.453
  20:44:37 STEP 1403 H= 350:45 +CPU= 44.532
  20:45:31 STEP 1404 H= 351:00 +CPU= 53.125
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001404
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001404
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001404
  20:46:18 STEP 1405 H= 351:15 +CPU= 46.933
  20:47:02 STEP 1406 H= 351:30 +CPU= 44.098
  20:47:49 STEP 1407 H= 351:45 +CPU= 46.014
  20:48:38 STEP 1408 H= 352:00 +CPU= 48.393
  20:49:29 STEP 1409 H= 352:15 +CPU= 50.598
  20:50:15 STEP 1410 H= 352:30 +CPU= 45.573
  20:51:03 STEP 1411 H= 352:45 +CPU= 48.362
  20:51:53 STEP 1412 H= 353:00 +CPU= 48.962
  20:52:42 STEP 1413 H= 353:15 +CPU= 48.798
  20:53:30 STEP 1414 H= 353:30 +CPU= 47.574
  20:54:19 STEP 1415 H= 353:45 +CPU= 47.550
  20:55:22 STEP 1416 H= 354:00 +CPU= 58.535
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001416
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001416
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001416
  20:56:13 STEP 1417 H= 354:15 +CPU= 50.495
  20:56:59 STEP 1418 H= 354:30 +CPU= 45.938
  20:57:44 STEP 1419 H= 354:45 +CPU= 43.852
  20:58:29 STEP 1420 H= 355:00 +CPU= 44.918
  20:59:11 STEP 1421 H= 355:15 +CPU= 41.928
  20:59:54 STEP 1422 H= 355:30 +CPU= 43.017
  21:00:38 STEP 1423 H= 355:45 +CPU= 43.123
  21:01:21 STEP 1424 H= 356:00 +CPU= 43.364
  21:02:06 STEP 1425 H= 356:15 +CPU= 43.860
  21:02:49 STEP 1426 H= 356:30 +CPU= 42.428
  21:03:32 STEP 1427 H= 356:45 +CPU= 43.230
  21:04:27 STEP 1428 H= 357:00 +CPU= 54.762
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001428
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001428
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001428
  21:05:11 STEP 1429 H= 357:15 +CPU= 43.205
  21:05:55 STEP 1430 H= 357:30 +CPU= 43.550
  21:06:37 STEP 1431 H= 357:45 +CPU= 42.253
  21:07:21 STEP 1432 H= 358:00 +CPU= 43.776
  21:08:04 STEP 1433 H= 358:15 +CPU= 42.403
  21:08:45 STEP 1434 H= 358:30 +CPU= 40.535
  21:09:26 STEP 1435 H= 358:45 +CPU= 40.156
  21:10:07 STEP 1436 H= 359:00 +CPU= 40.684
  21:10:48 STEP 1437 H= 359:15 +CPU= 41.467
  21:11:31 STEP 1438 H= 359:30 +CPU= 42.664
  21:12:15 STEP 1439 H= 359:45 +CPU= 42.967
  21:13:03 STEP 1440 H= 360:00 +CPU= 48.099
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.2019120408E-20      I       0.8640642774E-05      I
ICloud fractionI      0.0000000000E+00      I       0.2879069396E-01      I
I  Relat. Hum. I      0.4892115486E+00      I       0.4882978185E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.5031630338E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    320812900 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS458304.1 Mbytes/s, TOTAL TIME=    0.0146( 4.8%)
 21:13:03 STEP 1440 H= 360:00 +CPU= 48.099
 END CNT3
 NSTEP =  1440 CNT0   000000000  35.692  35.692  35.903   0.994               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        43246/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    43246 ************   122.1     0.0   122.0     0.0   282.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.99      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                 1562    9311.7    9311.7     18.93      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    1933.6    1933.6      3.62      0.00
  10 SCAN2M - PHYSICS                             1441   23869.6   23869.6     44.78      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1624.7    1624.7      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63     119.3     119.3      0.01      0.00
  14 SUINIF                                          1    7295.2    7295.2      0.01      0.00
  17 GRIDFPOS IN CNT4                              121      57.0      57.0      0.01      0.00
  18 SUSPECG                                         1    3758.8    3758.8      0.00      0.00
  19 SUSPEC                                          1    3768.5    3768.5      0.00      0.00
  24 SUGRIDU                                         1    2973.1    2973.1      0.00      0.00
  25 SPECRT                                          1     393.4     393.4      0.00      0.00
  26 SUGRIDF                                         1     160.2     160.2      0.00      0.00
  27 RESTART FILES - WRITING                        30    6553.1    6553.1      0.26      0.00
  28 RESTART FILES - READING                         1       0.0       0.0      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                               121   11160.4   11160.4      1.76      0.00
  31 POSDDH IN STEPO                                 7       1.2       1.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441   10638.3   10638.3     19.96      0.00
  39 SU0YOMB                                         1     499.8     499.8      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      39.3      39.3      0.07      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440    1063.2    1063.2      1.99      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440     255.0     255.0      0.48      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.8       0.8      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     747.1     747.1      3.04      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     430.2     430.2      1.68      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     154.5     154.5      0.60      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     235.7     235.7      0.96      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      63.7      63.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125     123.5     123.5      0.50      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      78.0      78.0      0.31      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125     186.5     186.5      0.76      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      81.6      81.6      0.32      0.00
 400 GSTATS                                     167734       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                156066       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    76819.3 CPU TIME   75406.2 VECTOR TIME    75406.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       16.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/steve/slots/12/ICMGGh7zg+001440
Moving to projects directory: /home/steve/slots/12/ICMSHh7zg+001440
Moving to projects directory: /home/steve/slots/12/ICMUAh7zg+001440
Adding to the zip: /home/steve/slots/12/NODE.001_01
Adding to the zip: /home/steve/slots/12/ifs.stat
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMUAh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMSHh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12284957/ICMGGh7zg+001404
Zipping up the final file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
21:16:32 (998): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1_r40521878_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 03:15:44 1536520 22440419 oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1 1,295,100 75,406 0.0582
08 Jun 2024 20:59:19 1536520 22440419 oifs_43r3_bl_a2ey_2016092300_20_1017_12284957_1 864,000 53,105 0.0615


©2024 climateprediction.net