climateprediction.net home page
Task 22439359

Task 22439359

Name oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1
Workunit 12285428
Created 7 Jun 2024, 19:21:04 UTC
Sent 7 Jun 2024, 20:28:48 UTC
Report deadline 6 Aug 2024, 20:28:48 UTC
Received 8 Jun 2024, 18:41:28 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1536520
Run time 21 hours 24 min 32 sec
CPU time 20 hours 56 min 24 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.23 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.45 MB
Peak swap size 5,980.81 MB
Peak disk usage 1,356.42 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
+000780
  02:12:04 STEP  781 H= 195:15 +CPU= 60.850
  02:13:02 STEP  782 H= 195:30 +CPU= 57.123
  02:13:59 STEP  783 H= 195:45 +CPU= 57.084
  02:14:59 STEP  784 H= 196:00 +CPU= 59.023
  02:15:58 STEP  785 H= 196:15 +CPU= 58.715
  02:16:59 STEP  786 H= 196:30 +CPU= 59.536
  02:18:00 STEP  787 H= 196:45 +CPU= 60.793
  02:19:00 STEP  788 H= 197:00 +CPU= 59.538
  02:19:59 STEP  789 H= 197:15 +CPU= 57.793
  02:20:56 STEP  790 H= 197:30 +CPU= 57.247
  02:21:54 STEP  791 H= 197:45 +CPU= 57.079
  02:23:11 STEP  792 H= 198:00 +CPU= 75.604
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000792
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000792
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000792
  02:24:09 STEP  793 H= 198:15 +CPU= 57.456
  02:25:07 STEP  794 H= 198:30 +CPU= 57.442
  02:26:02 STEP  795 H= 198:45 +CPU= 54.842
  02:27:00 STEP  796 H= 199:00 +CPU= 56.807
  02:27:57 STEP  797 H= 199:15 +CPU= 56.899
  02:28:57 STEP  798 H= 199:30 +CPU= 58.971
  02:29:54 STEP  799 H= 199:45 +CPU= 57.094
  02:30:56 STEP  800 H= 200:00 +CPU= 60.810
  02:31:53 STEP  801 H= 200:15 +CPU= 57.074
  02:32:51 STEP  802 H= 200:30 +CPU= 57.452
  02:33:52 STEP  803 H= 200:45 +CPU= 59.730
  02:35:02 STEP  804 H= 201:00 +CPU= 69.646
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000804
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000804
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000804
  02:35:59 STEP  805 H= 201:15 +CPU= 56.194
  02:36:57 STEP  806 H= 201:30 +CPU= 57.152
  02:37:53 STEP  807 H= 201:45 +CPU= 56.163
  02:38:54 STEP  808 H= 202:00 +CPU= 60.418
  02:39:55 STEP  809 H= 202:15 +CPU= 60.448
  02:40:55 STEP  810 H= 202:30 +CPU= 58.856
  02:41:55 STEP  811 H= 202:45 +CPU= 59.534
  02:42:53 STEP  812 H= 203:00 +CPU= 57.664
  02:43:50 STEP  813 H= 203:15 +CPU= 56.196
  02:44:46 STEP  814 H= 203:30 +CPU= 55.640
  02:45:42 STEP  815 H= 203:45 +CPU= 55.170
  02:46:59 STEP  816 H= 204:00 +CPU= 76.051
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000816
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000816
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000816
  02:47:56 STEP  817 H= 204:15 +CPU= 56.176
  02:48:56 STEP  818 H= 204:30 +CPU= 59.291
  02:49:56 STEP  819 H= 204:45 +CPU= 59.818
  02:50:53 STEP  820 H= 205:00 +CPU= 56.548
  02:51:53 STEP  821 H= 205:15 +CPU= 58.691
  02:52:52 STEP  822 H= 205:30 +CPU= 58.265
  02:53:49 STEP  823 H= 205:45 +CPU= 57.142
  02:54:51 STEP  824 H= 206:00 +CPU= 60.781
  02:55:52 STEP  825 H= 206:15 +CPU= 61.274
  02:56:51 STEP  826 H= 206:30 +CPU= 57.464
  02:57:52 STEP  827 H= 206:45 +CPU= 60.325
  02:59:02 STEP  828 H= 207:00 +CPU= 69.575
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000828
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000828
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000828
  03:00:00 STEP  829 H= 207:15 +CPU= 57.366
  03:01:00 STEP  830 H= 207:30 +CPU= 59.843
  03:01:59 STEP  831 H= 207:45 +CPU= 57.724
  03:02:58 STEP  832 H= 208:00 +CPU= 58.773
  03:03:57 STEP  833 H= 208:15 +CPU= 57.317
  03:04:54 STEP  834 H= 208:30 +CPU= 56.382
  03:05:50 STEP  835 H= 208:45 +CPU= 55.956
  03:06:48 STEP  836 H= 209:00 +CPU= 56.650
  03:07:45 STEP  837 H= 209:15 +CPU= 56.187
  03:08:44 STEP  838 H= 209:30 +CPU= 59.159
  03:09:43 STEP  839 H= 209:45 +CPU= 57.770
  03:10:53 STEP  840 H= 210:00 +CPU= 69.615
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000840
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000840
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000840
  03:11:53 STEP  841 H= 210:15 +CPU= 59.059
  03:12:53 STEP  842 H= 210:30 +CPU= 59.620
  03:13:53 STEP  843 H= 210:45 +CPU= 59.007
  03:14:52 STEP  844 H= 211:00 +CPU= 59.029
  03:15:50 STEP  845 H= 211:15 +CPU= 56.828
  03:16:49 STEP  846 H= 211:30 +CPU= 58.329
  03:17:46 STEP  847 H= 211:45 +CPU= 57.104
  03:18:44 STEP  848 H= 212:00 +CPU= 57.604
  03:19:43 STEP  849 H= 212:15 +CPU= 58.073
  03:20:42 STEP  850 H= 212:30 +CPU= 58.311
  03:21:41 STEP  851 H= 212:45 +CPU= 57.929
  03:22:50 STEP  852 H= 213:00 +CPU= 68.812
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000852
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000852
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000852
  03:23:47 STEP  853 H= 213:15 +CPU= 55.996
  03:24:45 STEP  854 H= 213:30 +CPU= 55.876
  03:25:41 STEP  855 H= 213:45 +CPU= 55.021
  03:26:37 STEP  856 H= 214:00 +CPU= 54.997
  03:27:32 STEP  857 H= 214:15 +CPU= 55.158
  03:28:31 STEP  858 H= 214:30 +CPU= 58.345
  03:29:33 STEP  859 H= 214:45 +CPU= 60.946
  03:30:31 STEP  860 H= 215:00 +CPU= 57.834
  03:31:27 STEP  861 H= 215:15 +CPU= 55.738
  03:32:25 STEP  862 H= 215:30 +CPU= 57.123
  03:33:25 STEP  863 H= 215:45 +CPU= 59.732
  03:34:46 STEP  864 H= 216:00 +CPU= 79.659
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000864
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000864
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000768
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000780
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000792
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000804
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000816
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000828
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000840
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000852
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000852
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_8.zip
Uploading the intermediate file: upload_file_8.zip
  03:35:44 STEP  865 H= 216:15 +CPU= 56.884
  03:36:40 STEP  866 H= 216:30 +CPU= 55.044
  03:37:36 STEP  867 H= 216:45 +CPU= 56.128
  03:38:34 STEP  868 H= 217:00 +CPU= 56.750
  03:39:31 STEP  869 H= 217:15 +CPU= 54.791
  03:40:28 STEP  870 H= 217:30 +CPU= 56.560
  03:41:27 STEP  871 H= 217:45 +CPU= 58.375
  03:42:25 STEP  872 H= 218:00 +CPU= 57.589
  03:43:24 STEP  873 H= 218:15 +CPU= 58.336
  03:44:23 STEP  874 H= 218:30 +CPU= 58.840
  03:45:21 STEP  875 H= 218:45 +CPU= 56.553
  03:46:31 STEP  876 H= 219:00 +CPU= 69.612
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000876
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000876
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000876
  03:47:27 STEP  877 H= 219:15 +CPU= 55.648
  03:48:24 STEP  878 H= 219:30 +CPU= 56.166
  03:49:25 STEP  879 H= 219:45 +CPU= 60.333
  03:50:24 STEP  880 H= 220:00 +CPU= 58.739
  03:51:24 STEP  881 H= 220:15 +CPU= 58.920
  03:52:23 STEP  882 H= 220:30 +CPU= 58.772
  03:53:21 STEP  883 H= 220:45 +CPU= 57.550
  03:54:22 STEP  884 H= 221:00 +CPU= 60.278
  03:55:24 STEP  885 H= 221:15 +CPU= 61.499
  03:56:23 STEP  886 H= 221:30 +CPU= 58.322
  03:57:24 STEP  887 H= 221:45 +CPU= 59.839
  03:58:40 STEP  888 H= 222:00 +CPU= 75.242
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000888
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000888
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000888
  03:59:37 STEP  889 H= 222:15 +CPU= 56.773
  04:00:37 STEP  890 H= 222:30 +CPU= 59.310
  04:01:36 STEP  891 H= 222:45 +CPU= 58.015
  04:02:35 STEP  892 H= 223:00 +CPU= 58.955
  04:03:33 STEP  893 H= 223:15 +CPU= 57.457
  04:04:29 STEP  894 H= 223:30 +CPU= 54.767
  04:05:24 STEP  895 H= 223:45 +CPU= 54.526
  04:06:20 STEP  896 H= 224:00 +CPU= 55.529
  04:07:15 STEP  897 H= 224:15 +CPU= 52.768
  04:08:12 STEP  898 H= 224:30 +CPU= 55.854
  04:09:07 STEP  899 H= 224:45 +CPU= 54.607
  04:10:18 STEP  900 H= 225:00 +CPU= 70.342
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000900
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000900
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000900
  04:11:13 STEP  901 H= 225:15 +CPU= 54.070
  04:12:09 STEP  902 H= 225:30 +CPU= 55.776
  04:13:05 STEP  903 H= 225:45 +CPU= 55.221
  04:14:01 STEP  904 H= 226:00 +CPU= 55.832
  04:14:55 STEP  905 H= 226:15 +CPU= 53.437
  04:15:51 STEP  906 H= 226:30 +CPU= 55.418
  04:16:46 STEP  907 H= 226:45 +CPU= 54.636
  04:17:44 STEP  908 H= 227:00 +CPU= 57.145
  04:18:40 STEP  909 H= 227:15 +CPU= 55.235
  04:19:35 STEP  910 H= 227:30 +CPU= 54.640
  04:20:31 STEP  911 H= 227:45 +CPU= 55.372
  04:21:48 STEP  912 H= 228:00 +CPU= 76.039
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000912
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000912
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000912
  04:22:42 STEP  913 H= 228:15 +CPU= 54.030
  04:23:36 STEP  914 H= 228:30 +CPU= 53.021
  04:24:30 STEP  915 H= 228:45 +CPU= 53.867
  04:25:25 STEP  916 H= 229:00 +CPU= 54.501
  04:26:21 STEP  917 H= 229:15 +CPU= 54.919
  04:27:17 STEP  918 H= 229:30 +CPU= 55.317
  04:28:10 STEP  919 H= 229:45 +CPU= 53.222
  04:29:04 STEP  920 H= 230:00 +CPU= 53.671
  04:30:01 STEP  921 H= 230:15 +CPU= 55.871
  04:30:56 STEP  922 H= 230:30 +CPU= 54.535
  04:31:52 STEP  923 H= 230:45 +CPU= 55.227
  04:33:02 STEP  924 H= 231:00 +CPU= 69.597
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000924
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000924
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000924
  04:33:57 STEP  925 H= 231:15 +CPU= 54.698
  04:34:52 STEP  926 H= 231:30 +CPU= 53.746
  04:35:47 STEP  927 H= 231:45 +CPU= 54.467
  04:36:43 STEP  928 H= 232:00 +CPU= 55.815
  04:37:39 STEP  929 H= 232:15 +CPU= 55.082
  04:38:34 STEP  930 H= 232:30 +CPU= 54.139
  04:39:30 STEP  931 H= 232:45 +CPU= 54.448
  04:40:25 STEP  932 H= 233:00 +CPU= 54.985
  04:41:21 STEP  933 H= 233:15 +CPU= 55.111
  04:42:17 STEP  934 H= 233:30 +CPU= 55.363
  04:43:12 STEP  935 H= 233:45 +CPU= 54.479
  04:44:20 STEP  936 H= 234:00 +CPU= 67.146
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000936
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000936
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000936
  04:45:15 STEP  937 H= 234:15 +CPU= 54.403
  04:46:09 STEP  938 H= 234:30 +CPU= 53.996
  04:47:04 STEP  939 H= 234:45 +CPU= 53.977
  04:48:00 STEP  940 H= 235:00 +CPU= 56.182
  04:48:54 STEP  941 H= 235:15 +CPU= 53.578
  04:49:50 STEP  942 H= 235:30 +CPU= 55.028
  04:50:47 STEP  943 H= 235:45 +CPU= 56.291
  04:51:45 STEP  944 H= 236:00 +CPU= 57.592
  04:52:40 STEP  945 H= 236:15 +CPU= 54.144
  04:53:35 STEP  946 H= 236:30 +CPU= 54.449
  04:54:31 STEP  947 H= 236:45 +CPU= 55.589
  04:55:41 STEP  948 H= 237:00 +CPU= 69.907
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000948
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000948
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000948
  04:56:36 STEP  949 H= 237:15 +CPU= 53.900
  04:57:31 STEP  950 H= 237:30 +CPU= 52.839
  04:58:27 STEP  951 H= 237:45 +CPU= 55.697
  04:59:24 STEP  952 H= 238:00 +CPU= 55.933
  05:00:19 STEP  953 H= 238:15 +CPU= 54.612
  05:01:14 STEP  954 H= 238:30 +CPU= 54.288
  05:02:10 STEP  955 H= 238:45 +CPU= 54.943
  05:03:06 STEP  956 H= 239:00 +CPU= 55.554
  05:04:02 STEP  957 H= 239:15 +CPU= 55.819
  05:04:57 STEP  958 H= 239:30 +CPU= 54.749
  05:05:53 STEP  959 H= 239:45 +CPU= 54.723
  05:07:09 STEP  960 H= 240:00 +CPU= 75.115
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000960
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000960
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000864
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000876
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000888
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000900
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000912
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000924
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000936
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000948
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000948
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  05:08:05 STEP  961 H= 240:15 +CPU= 54.368
  05:09:00 STEP  962 H= 240:30 +CPU= 54.503
  05:09:55 STEP  963 H= 240:45 +CPU= 55.033
  05:10:51 STEP  964 H= 241:00 +CPU= 54.640
  05:11:46 STEP  965 H= 241:15 +CPU= 53.178
  05:12:42 STEP  966 H= 241:30 +CPU= 55.426
  05:13:38 STEP  967 H= 241:45 +CPU= 55.597
  05:14:34 STEP  968 H= 242:00 +CPU= 55.129
  05:15:29 STEP  969 H= 242:15 +CPU= 54.575
  05:16:25 STEP  970 H= 242:30 +CPU= 56.015
  05:17:20 STEP  971 H= 242:45 +CPU= 54.267
  05:18:30 STEP  972 H= 243:00 +CPU= 69.315
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000972
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000972
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000972
  05:19:25 STEP  973 H= 243:15 +CPU= 54.544
  05:20:21 STEP  974 H= 243:30 +CPU= 55.424
  05:21:17 STEP  975 H= 243:45 +CPU= 55.818
  05:22:13 STEP  976 H= 244:00 +CPU= 55.407
  05:23:10 STEP  977 H= 244:15 +CPU= 55.816
  05:24:04 STEP  978 H= 244:30 +CPU= 53.451
  05:25:00 STEP  979 H= 244:45 +CPU= 55.478
  05:25:55 STEP  980 H= 245:00 +CPU= 54.506
  05:26:50 STEP  981 H= 245:15 +CPU= 55.112
  05:27:46 STEP  982 H= 245:30 +CPU= 54.819
  05:28:43 STEP  983 H= 245:45 +CPU= 56.988
  05:29:52 STEP  984 H= 246:00 +CPU= 67.697
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000984
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000984
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000984
  05:30:48 STEP  985 H= 246:15 +CPU= 55.162
  05:31:44 STEP  986 H= 246:30 +CPU= 56.291
  05:32:40 STEP  987 H= 246:45 +CPU= 55.438
  05:33:36 STEP  988 H= 247:00 +CPU= 55.012
  05:34:30 STEP  989 H= 247:15 +CPU= 54.172
  05:35:24 STEP  990 H= 247:30 +CPU= 53.353
  05:36:20 STEP  991 H= 247:45 +CPU= 55.443
  05:37:17 STEP  992 H= 248:00 +CPU= 56.711
  05:38:13 STEP  993 H= 248:15 +CPU= 55.298
  05:39:10 STEP  994 H= 248:30 +CPU= 55.918
  05:40:06 STEP  995 H= 248:45 +CPU= 55.191
  05:41:15 STEP  996 H= 249:00 +CPU= 68.558
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+000996
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+000996
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+000996
  05:42:10 STEP  997 H= 249:15 +CPU= 54.374
  05:43:06 STEP  998 H= 249:30 +CPU= 55.809
  05:44:01 STEP  999 H= 249:45 +CPU= 54.188
  05:44:59 STEP 1000 H= 250:00 +CPU= 57.463
  05:45:53 STEP 1001 H= 250:15 +CPU= 53.502
  05:46:48 STEP 1002 H= 250:30 +CPU= 54.826
  05:47:44 STEP 1003 H= 250:45 +CPU= 54.976
  05:48:41 STEP 1004 H= 251:00 +CPU= 56.362
  05:49:35 STEP 1005 H= 251:15 +CPU= 54.045
  05:50:30 STEP 1006 H= 251:30 +CPU= 54.348
  05:51:26 STEP 1007 H= 251:45 +CPU= 54.781
  05:52:42 STEP 1008 H= 252:00 +CPU= 75.247
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001008
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001008
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001008
  05:53:36 STEP 1009 H= 252:15 +CPU= 53.533
  05:54:31 STEP 1010 H= 252:30 +CPU= 54.323
  05:55:27 STEP 1011 H= 252:45 +CPU= 55.478
  05:56:22 STEP 1012 H= 253:00 +CPU= 54.970
  05:57:18 STEP 1013 H= 253:15 +CPU= 55.239
  05:58:15 STEP 1014 H= 253:30 +CPU= 56.136
  05:59:11 STEP 1015 H= 253:45 +CPU= 55.470
  06:00:07 STEP 1016 H= 254:00 +CPU= 56.046
  06:01:03 STEP 1017 H= 254:15 +CPU= 54.731
  06:01:59 STEP 1018 H= 254:30 +CPU= 55.403
  06:02:54 STEP 1019 H= 254:45 +CPU= 54.861
  06:04:02 STEP 1020 H= 255:00 +CPU= 67.600
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001020
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001020
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001020
  06:04:56 STEP 1021 H= 255:15 +CPU= 53.035
  06:05:51 STEP 1022 H= 255:30 +CPU= 54.568
  06:06:47 STEP 1023 H= 255:45 +CPU= 55.927
  06:07:43 STEP 1024 H= 256:00 +CPU= 55.093
  06:08:38 STEP 1025 H= 256:15 +CPU= 54.105
  06:09:34 STEP 1026 H= 256:30 +CPU= 54.632
  06:10:28 STEP 1027 H= 256:45 +CPU= 54.231
  06:11:24 STEP 1028 H= 257:00 +CPU= 55.224
  06:12:19 STEP 1029 H= 257:15 +CPU= 54.494
  06:13:15 STEP 1030 H= 257:30 +CPU= 54.652
  06:14:08 STEP 1031 H= 257:45 +CPU= 53.172
  06:15:19 STEP 1032 H= 258:00 +CPU= 68.198
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001032
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001032
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001032
  06:16:14 STEP 1033 H= 258:15 +CPU= 54.254
  06:17:11 STEP 1034 H= 258:30 +CPU= 55.920
  06:18:08 STEP 1035 H= 258:45 +CPU= 56.891
  06:19:04 STEP 1036 H= 259:00 +CPU= 55.638
  06:20:00 STEP 1037 H= 259:15 +CPU= 55.058
  06:20:58 STEP 1038 H= 259:30 +CPU= 52.370
  06:21:57 STEP 1039 H= 259:45 +CPU= 52.038
  06:22:57 STEP 1040 H= 260:00 +CPU= 54.780
  06:23:54 STEP 1041 H= 260:15 +CPU= 55.696
  06:24:53 STEP 1042 H= 260:30 +CPU= 54.526
  06:25:51 STEP 1043 H= 260:45 +CPU= 55.237
  06:27:01 STEP 1044 H= 261:00 +CPU= 67.477
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001044
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001044
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001044
  06:28:01 STEP 1045 H= 261:15 +CPU= 54.936
  06:28:57 STEP 1046 H= 261:30 +CPU= 52.586
  06:29:54 STEP 1047 H= 261:45 +CPU= 54.878
  06:30:52 STEP 1048 H= 262:00 +CPU= 55.186
  06:31:52 STEP 1049 H= 262:15 +CPU= 56.114
  06:32:51 STEP 1050 H= 262:30 +CPU= 55.494
  06:33:52 STEP 1051 H= 262:45 +CPU= 56.073
  06:34:51 STEP 1052 H= 263:00 +CPU= 55.062
  06:35:46 STEP 1053 H= 263:15 +CPU= 54.238
  06:36:42 STEP 1054 H= 263:30 +CPU= 55.217
  06:37:39 STEP 1055 H= 263:45 +CPU= 55.026
  06:38:57 STEP 1056 H= 264:00 +CPU= 73.016
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001056
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001056
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000960
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000972
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000984
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+000996
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001008
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001020
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001032
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001044
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001044
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_10.zip
Uploading the intermediate file: upload_file_10.zip
  06:39:55 STEP 1057 H= 264:15 +CPU= 52.879
  06:40:51 STEP 1058 H= 264:30 +CPU= 54.407
  06:41:49 STEP 1059 H= 264:45 +CPU= 54.933
  06:42:47 STEP 1060 H= 265:00 +CPU= 54.685
  06:43:43 STEP 1061 H= 265:15 +CPU= 52.440
  06:44:41 STEP 1062 H= 265:30 +CPU= 54.950
  06:45:38 STEP 1063 H= 265:45 +CPU= 54.796
  06:46:36 STEP 1064 H= 266:00 +CPU= 56.111
  06:47:33 STEP 1065 H= 266:15 +CPU= 54.638
  06:48:31 STEP 1066 H= 266:30 +CPU= 55.274
  06:49:30 STEP 1067 H= 266:45 +CPU= 55.189
  06:50:40 STEP 1068 H= 267:00 +CPU= 68.543
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001068
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001068
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001068
  06:51:36 STEP 1069 H= 267:15 +CPU= 54.962
  06:52:35 STEP 1070 H= 267:30 +CPU= 56.476
  06:53:31 STEP 1071 H= 267:45 +CPU= 55.003
  06:54:28 STEP 1072 H= 268:00 +CPU= 55.603
  06:55:23 STEP 1073 H= 268:15 +CPU= 53.690
  06:56:21 STEP 1074 H= 268:30 +CPU= 56.505
  06:57:18 STEP 1075 H= 268:45 +CPU= 55.468
  06:58:18 STEP 1076 H= 269:00 +CPU= 56.118
  06:59:15 STEP 1077 H= 269:15 +CPU= 55.370
  07:00:16 STEP 1078 H= 269:30 +CPU= 55.860
  07:01:14 STEP 1079 H= 269:45 +CPU= 54.797
  07:02:27 STEP 1080 H= 270:00 +CPU= 69.784
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001080
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001080
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001080
  07:03:23 STEP 1081 H= 270:15 +CPU= 54.055
  07:04:21 STEP 1082 H= 270:30 +CPU= 55.447
  07:05:16 STEP 1083 H= 270:45 +CPU= 54.547
  07:06:12 STEP 1084 H= 271:00 +CPU= 54.938
  07:07:08 STEP 1085 H= 271:15 +CPU= 55.200
  07:08:02 STEP 1086 H= 271:30 +CPU= 53.973
  07:08:57 STEP 1087 H= 271:45 +CPU= 54.600
  07:09:53 STEP 1088 H= 272:00 +CPU= 55.361
  07:10:49 STEP 1089 H= 272:15 +CPU= 55.547
  07:11:45 STEP 1090 H= 272:30 +CPU= 55.022
  07:12:41 STEP 1091 H= 272:45 +CPU= 56.025
  07:13:51 STEP 1092 H= 273:00 +CPU= 69.132
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001092
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001092
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001092
  07:14:46 STEP 1093 H= 273:15 +CPU= 54.051
  07:15:41 STEP 1094 H= 273:30 +CPU= 54.531
  07:16:37 STEP 1095 H= 273:45 +CPU= 55.662
  07:17:34 STEP 1096 H= 274:00 +CPU= 56.033
  07:18:29 STEP 1097 H= 274:15 +CPU= 54.630
  07:19:24 STEP 1098 H= 274:30 +CPU= 55.134
  07:20:20 STEP 1099 H= 274:45 +CPU= 55.278
  07:21:16 STEP 1100 H= 275:00 +CPU= 55.029
  07:22:11 STEP 1101 H= 275:15 +CPU= 54.569
  07:23:07 STEP 1102 H= 275:30 +CPU= 55.306
  07:24:02 STEP 1103 H= 275:45 +CPU= 55.237
  07:25:20 STEP 1104 H= 276:00 +CPU= 74.545
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001104
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001104
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001104
  07:26:17 STEP 1105 H= 276:15 +CPU= 53.136
  07:27:15 STEP 1106 H= 276:30 +CPU= 55.373
  07:28:12 STEP 1107 H= 276:45 +CPU= 56.018
  07:29:11 STEP 1108 H= 277:00 +CPU= 55.238
  07:30:10 STEP 1109 H= 277:15 +CPU= 55.696
  07:31:08 STEP 1110 H= 277:30 +CPU= 55.431
  07:32:06 STEP 1111 H= 277:45 +CPU= 55.798
  07:33:06 STEP 1112 H= 278:00 +CPU= 57.029
  07:34:03 STEP 1113 H= 278:15 +CPU= 54.840
  07:35:01 STEP 1114 H= 278:30 +CPU= 54.656
  07:35:59 STEP 1115 H= 278:45 +CPU= 55.220
  07:37:12 STEP 1116 H= 279:00 +CPU= 69.052
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001116
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001116
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001116
  07:38:10 STEP 1117 H= 279:15 +CPU= 55.794
  07:39:08 STEP 1118 H= 279:30 +CPU= 55.753
  07:40:04 STEP 1119 H= 279:45 +CPU= 55.020
  07:41:02 STEP 1120 H= 280:00 +CPU= 55.303
  07:42:00 STEP 1121 H= 280:15 +CPU= 55.287
  07:42:58 STEP 1122 H= 280:30 +CPU= 55.422
  07:43:54 STEP 1123 H= 280:45 +CPU= 54.718
  07:44:48 STEP 1124 H= 281:00 +CPU= 54.162
  07:45:50 STEP 1125 H= 281:15 +CPU= 54.242
  07:46:52 STEP 1126 H= 281:30 +CPU= 54.377
  07:47:52 STEP 1127 H= 281:45 +CPU= 53.081
  07:49:06 STEP 1128 H= 282:00 +CPU= 66.973
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001128
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001128
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001128
  07:50:03 STEP 1129 H= 282:15 +CPU= 54.395
  07:50:59 STEP 1130 H= 282:30 +CPU= 54.531
  07:51:56 STEP 1131 H= 282:45 +CPU= 54.605
  07:52:54 STEP 1132 H= 283:00 +CPU= 55.341
  07:53:50 STEP 1133 H= 283:15 +CPU= 54.058
  07:54:48 STEP 1134 H= 283:30 +CPU= 54.794
  07:55:44 STEP 1135 H= 283:45 +CPU= 54.154
  07:56:44 STEP 1136 H= 284:00 +CPU= 56.592
  07:57:42 STEP 1137 H= 284:15 +CPU= 54.903
  07:58:40 STEP 1138 H= 284:30 +CPU= 55.168
  07:59:35 STEP 1139 H= 284:45 +CPU= 55.035
  08:00:44 STEP 1140 H= 285:00 +CPU= 68.089
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001140
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001140
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001140
  08:01:40 STEP 1141 H= 285:15 +CPU= 55.145
  08:02:36 STEP 1142 H= 285:30 +CPU= 54.263
  08:03:33 STEP 1143 H= 285:45 +CPU= 54.945
  08:04:28 STEP 1144 H= 286:00 +CPU= 54.485
  08:05:24 STEP 1145 H= 286:15 +CPU= 55.582
  08:06:20 STEP 1146 H= 286:30 +CPU= 54.933
  08:07:15 STEP 1147 H= 286:45 +CPU= 54.695
  08:08:13 STEP 1148 H= 287:00 +CPU= 57.067
  08:09:09 STEP 1149 H= 287:15 +CPU= 55.491
  08:10:03 STEP 1150 H= 287:30 +CPU= 54.417
  08:10:58 STEP 1151 H= 287:45 +CPU= 54.156
  08:12:13 STEP 1152 H= 288:00 +CPU= 74.538
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001152
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001152
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001056
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001068
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001080
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001092
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001104
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001116
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001128
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001140
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001140
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_11.zip
Uploading the intermediate file: upload_file_11.zip
  08:13:09 STEP 1153 H= 288:15 +CPU= 53.971
  08:14:05 STEP 1154 H= 288:30 +CPU= 55.031
  08:15:00 STEP 1155 H= 288:45 +CPU= 55.158
  08:15:58 STEP 1156 H= 289:00 +CPU= 56.529
  08:16:53 STEP 1157 H= 289:15 +CPU= 53.943
  08:17:49 STEP 1158 H= 289:30 +CPU= 54.488
  08:18:44 STEP 1159 H= 289:45 +CPU= 54.779
  08:19:42 STEP 1160 H= 290:00 +CPU= 56.647
  08:20:38 STEP 1161 H= 290:15 +CPU= 56.009
  08:21:34 STEP 1162 H= 290:30 +CPU= 55.175
  08:22:32 STEP 1163 H= 290:45 +CPU= 57.662
  08:23:43 STEP 1164 H= 291:00 +CPU= 69.773
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001164
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001164
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001164
  08:24:37 STEP 1165 H= 291:15 +CPU= 53.838
  08:25:33 STEP 1166 H= 291:30 +CPU= 55.217
  08:26:28 STEP 1167 H= 291:45 +CPU= 54.239
  08:27:23 STEP 1168 H= 292:00 +CPU= 55.328
  08:28:17 STEP 1169 H= 292:15 +CPU= 53.053
  08:29:13 STEP 1170 H= 292:30 +CPU= 55.106
  08:30:07 STEP 1171 H= 292:45 +CPU= 53.685
  08:31:03 STEP 1172 H= 293:00 +CPU= 55.885
  08:31:59 STEP 1173 H= 293:15 +CPU= 54.759
  08:32:54 STEP 1174 H= 293:30 +CPU= 54.906
  08:33:50 STEP 1175 H= 293:45 +CPU= 55.428
  08:34:59 STEP 1176 H= 294:00 +CPU= 68.256
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001176
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001176
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001176
  08:35:54 STEP 1177 H= 294:15 +CPU= 54.101
  08:36:49 STEP 1178 H= 294:30 +CPU= 54.842
  08:37:43 STEP 1179 H= 294:45 +CPU= 53.396
  08:38:39 STEP 1180 H= 295:00 +CPU= 55.644
  08:39:37 STEP 1181 H= 295:15 +CPU= 57.306
  08:40:32 STEP 1182 H= 295:30 +CPU= 55.014
  08:41:27 STEP 1183 H= 295:45 +CPU= 54.254
  08:42:24 STEP 1184 H= 296:00 +CPU= 56.839
  08:43:19 STEP 1185 H= 296:15 +CPU= 54.063
  08:44:15 STEP 1186 H= 296:30 +CPU= 54.965
  08:45:10 STEP 1187 H= 296:45 +CPU= 54.839
  08:46:20 STEP 1188 H= 297:00 +CPU= 69.648
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001188
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001188
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001188
  08:47:14 STEP 1189 H= 297:15 +CPU= 53.586
  08:48:10 STEP 1190 H= 297:30 +CPU= 55.136
  08:49:08 STEP 1191 H= 297:45 +CPU= 57.491
  08:50:04 STEP 1192 H= 298:00 +CPU= 55.318
  08:51:00 STEP 1193 H= 298:15 +CPU= 55.519
  08:51:55 STEP 1194 H= 298:30 +CPU= 54.249
  08:52:50 STEP 1195 H= 298:45 +CPU= 54.935
  08:53:47 STEP 1196 H= 299:00 +CPU= 55.860
  08:54:41 STEP 1197 H= 299:15 +CPU= 53.316
  08:55:37 STEP 1198 H= 299:30 +CPU= 56.178
  08:56:33 STEP 1199 H= 299:45 +CPU= 54.869
  08:57:50 STEP 1200 H= 300:00 +CPU= 76.956
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001200
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001200
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001200
  08:58:45 STEP 1201 H= 300:15 +CPU= 53.690
  08:59:40 STEP 1202 H= 300:30 +CPU= 54.338
  09:00:35 STEP 1203 H= 300:45 +CPU= 54.728
  09:01:31 STEP 1204 H= 301:00 +CPU= 55.824
  09:02:25 STEP 1205 H= 301:15 +CPU= 53.735
  09:03:22 STEP 1206 H= 301:30 +CPU= 55.755
  09:04:17 STEP 1207 H= 301:45 +CPU= 54.825
  09:05:13 STEP 1208 H= 302:00 +CPU= 55.128
  09:06:09 STEP 1209 H= 302:15 +CPU= 55.680
  09:07:05 STEP 1210 H= 302:30 +CPU= 55.709
  09:08:00 STEP 1211 H= 302:45 +CPU= 54.433
  09:09:11 STEP 1212 H= 303:00 +CPU= 69.975
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001212
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001212
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001212
  09:10:08 STEP 1213 H= 303:15 +CPU= 54.499
  09:11:06 STEP 1214 H= 303:30 +CPU= 55.152
  09:12:05 STEP 1215 H= 303:45 +CPU= 53.498
  09:13:05 STEP 1216 H= 304:00 +CPU= 54.151
  09:14:03 STEP 1217 H= 304:15 +CPU= 54.126
  09:15:02 STEP 1218 H= 304:30 +CPU= 53.873
  09:16:00 STEP 1219 H= 304:45 +CPU= 54.337
  09:16:55 STEP 1220 H= 305:00 +CPU= 54.744
  09:17:50 STEP 1221 H= 305:15 +CPU= 54.485
  09:18:45 STEP 1222 H= 305:30 +CPU= 54.527
  09:19:39 STEP 1223 H= 305:45 +CPU= 53.481
  09:20:48 STEP 1224 H= 306:00 +CPU= 68.112
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001224
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001224
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001224
  09:21:43 STEP 1225 H= 306:15 +CPU= 54.528
  09:22:40 STEP 1226 H= 306:30 +CPU= 56.538
  09:23:35 STEP 1227 H= 306:45 +CPU= 54.505
  09:24:30 STEP 1228 H= 307:00 +CPU= 54.848
  09:25:26 STEP 1229 H= 307:15 +CPU= 55.331
  09:26:23 STEP 1230 H= 307:30 +CPU= 55.868
  09:27:19 STEP 1231 H= 307:45 +CPU= 55.355
  09:28:16 STEP 1232 H= 308:00 +CPU= 57.104
  09:29:13 STEP 1233 H= 308:15 +CPU= 56.008
  09:30:10 STEP 1234 H= 308:30 +CPU= 56.317
  09:31:06 STEP 1235 H= 308:45 +CPU= 55.484
  09:32:15 STEP 1236 H= 309:00 +CPU= 68.499
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001236
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001236
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001236
  09:33:11 STEP 1237 H= 309:15 +CPU= 55.127
  09:34:06 STEP 1238 H= 309:30 +CPU= 54.521
  09:35:02 STEP 1239 H= 309:45 +CPU= 54.001
  09:35:57 STEP 1240 H= 310:00 +CPU= 53.766
  09:36:52 STEP 1241 H= 310:15 +CPU= 54.518
  09:37:47 STEP 1242 H= 310:30 +CPU= 54.810
  09:38:43 STEP 1243 H= 310:45 +CPU= 55.232
  09:39:38 STEP 1244 H= 311:00 +CPU= 54.630
  09:40:34 STEP 1245 H= 311:15 +CPU= 55.130
  09:41:30 STEP 1246 H= 311:30 +CPU= 55.862
  09:42:25 STEP 1247 H= 311:45 +CPU= 54.361
  09:43:42 STEP 1248 H= 312:00 +CPU= 75.876
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001248
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001248
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001152
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001164
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001176
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001188
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001200
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001212
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001224
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001236
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001236
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_12.zip
Uploading the intermediate file: upload_file_12.zip
  09:44:37 STEP 1249 H= 312:15 +CPU= 53.131
  09:45:32 STEP 1250 H= 312:30 +CPU= 54.839
  09:46:27 STEP 1251 H= 312:45 +CPU= 54.864
  09:47:23 STEP 1252 H= 313:00 +CPU= 54.902
  09:48:18 STEP 1253 H= 313:15 +CPU= 53.332
  09:49:15 STEP 1254 H= 313:30 +CPU= 55.660
  09:50:11 STEP 1255 H= 313:45 +CPU= 55.045
  09:51:06 STEP 1256 H= 314:00 +CPU= 54.990
  09:52:03 STEP 1257 H= 314:15 +CPU= 55.993
  09:52:59 STEP 1258 H= 314:30 +CPU= 55.353
  09:53:54 STEP 1259 H= 314:45 +CPU= 54.596
  09:55:05 STEP 1260 H= 315:00 +CPU= 69.774
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001260
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001260
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001260
  09:55:59 STEP 1261 H= 315:15 +CPU= 54.115
  09:56:56 STEP 1262 H= 315:30 +CPU= 56.004
  09:57:52 STEP 1263 H= 315:45 +CPU= 55.583
  09:58:48 STEP 1264 H= 316:00 +CPU= 55.282
  09:59:42 STEP 1265 H= 316:15 +CPU= 54.098
  10:00:38 STEP 1266 H= 316:30 +CPU= 55.008
  10:01:34 STEP 1267 H= 316:45 +CPU= 55.104
  10:02:30 STEP 1268 H= 317:00 +CPU= 56.143
  10:03:25 STEP 1269 H= 317:15 +CPU= 54.222
  10:04:19 STEP 1270 H= 317:30 +CPU= 53.825
  10:05:14 STEP 1271 H= 317:45 +CPU= 54.246
  10:06:24 STEP 1272 H= 318:00 +CPU= 68.851
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001272
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001272
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001272
  10:07:18 STEP 1273 H= 318:15 +CPU= 53.824
  10:08:14 STEP 1274 H= 318:30 +CPU= 55.669
  10:09:10 STEP 1275 H= 318:45 +CPU= 54.596
  10:10:08 STEP 1276 H= 319:00 +CPU= 58.044
  10:11:04 STEP 1277 H= 319:15 +CPU= 55.337
  10:12:00 STEP 1278 H= 319:30 +CPU= 55.612
  10:12:56 STEP 1279 H= 319:45 +CPU= 55.198
  10:13:52 STEP 1280 H= 320:00 +CPU= 55.724
  10:14:47 STEP 1281 H= 320:15 +CPU= 54.213
  10:15:41 STEP 1282 H= 320:30 +CPU= 53.344
  10:16:37 STEP 1283 H= 320:45 +CPU= 55.928
  10:17:47 STEP 1284 H= 321:00 +CPU= 68.909
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001284
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001284
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001284
  10:18:42 STEP 1285 H= 321:15 +CPU= 54.220
  10:19:36 STEP 1286 H= 321:30 +CPU= 54.226
  10:20:33 STEP 1287 H= 321:45 +CPU= 56.202
  10:21:29 STEP 1288 H= 322:00 +CPU= 55.451
  10:22:25 STEP 1289 H= 322:15 +CPU= 54.745
  10:23:21 STEP 1290 H= 322:30 +CPU= 56.255
  10:24:17 STEP 1291 H= 322:45 +CPU= 55.102
  10:25:14 STEP 1292 H= 323:00 +CPU= 56.810
  10:26:10 STEP 1293 H= 323:15 +CPU= 55.252
  10:27:06 STEP 1294 H= 323:30 +CPU= 55.913
  10:28:02 STEP 1295 H= 323:45 +CPU= 55.439
  10:29:19 STEP 1296 H= 324:00 +CPU= 75.883
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001296
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001296
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001296
  10:30:14 STEP 1297 H= 324:15 +CPU= 54.744
  10:31:10 STEP 1298 H= 324:30 +CPU= 55.479
  10:32:05 STEP 1299 H= 324:45 +CPU= 54.331
  10:33:01 STEP 1300 H= 325:00 +CPU= 55.388
  10:33:56 STEP 1301 H= 325:15 +CPU= 54.659
  10:34:52 STEP 1302 H= 325:30 +CPU= 55.152
  10:35:47 STEP 1303 H= 325:45 +CPU= 54.751
  10:36:43 STEP 1304 H= 326:00 +CPU= 55.130
  10:37:38 STEP 1305 H= 326:15 +CPU= 54.182
  10:38:33 STEP 1306 H= 326:30 +CPU= 54.357
  10:39:27 STEP 1307 H= 326:45 +CPU= 53.688
  10:40:36 STEP 1308 H= 327:00 +CPU= 68.946
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001308
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001308
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001308
  10:41:31 STEP 1309 H= 327:15 +CPU= 54.285
  10:42:27 STEP 1310 H= 327:30 +CPU= 54.945
  10:43:21 STEP 1311 H= 327:45 +CPU= 53.567
  10:44:16 STEP 1312 H= 328:00 +CPU= 54.576
  10:45:10 STEP 1313 H= 328:15 +CPU= 53.459
  10:46:04 STEP 1314 H= 328:30 +CPU= 52.974
  10:47:00 STEP 1315 H= 328:45 +CPU= 54.139
  10:47:57 STEP 1316 H= 329:00 +CPU= 56.338
  10:48:52 STEP 1317 H= 329:15 +CPU= 55.012
  10:49:48 STEP 1318 H= 329:30 +CPU= 55.140
  10:50:43 STEP 1319 H= 329:45 +CPU= 54.074
  10:51:52 STEP 1320 H= 330:00 +CPU= 68.564
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001320
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001320
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001320
  10:52:47 STEP 1321 H= 330:15 +CPU= 54.591
  10:53:42 STEP 1322 H= 330:30 +CPU= 54.572
  10:54:37 STEP 1323 H= 330:45 +CPU= 54.203
  10:55:32 STEP 1324 H= 331:00 +CPU= 54.958
  10:56:31 STEP 1325 H= 331:15 +CPU= 53.425
  10:57:28 STEP 1326 H= 331:30 +CPU= 54.524
  10:58:26 STEP 1327 H= 331:45 +CPU= 55.277
  10:59:23 STEP 1328 H= 332:00 +CPU= 55.541
  11:00:21 STEP 1329 H= 332:15 +CPU= 54.982
  11:01:17 STEP 1330 H= 332:30 +CPU= 54.340
  11:02:18 STEP 1331 H= 332:45 +CPU= 53.798
  11:03:34 STEP 1332 H= 333:00 +CPU= 66.046
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001332
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001332
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001332
  11:04:37 STEP 1333 H= 333:15 +CPU= 54.598
  11:05:38 STEP 1334 H= 333:30 +CPU= 54.393
  11:06:41 STEP 1335 H= 333:45 +CPU= 56.273
  11:07:43 STEP 1336 H= 334:00 +CPU= 56.628
  11:08:45 STEP 1337 H= 334:15 +CPU= 55.819
  11:09:45 STEP 1338 H= 334:30 +CPU= 55.115
  11:10:48 STEP 1339 H= 334:45 +CPU= 54.751
  11:11:54 STEP 1340 H= 335:00 +CPU= 56.754
  11:12:58 STEP 1341 H= 335:15 +CPU= 55.236
  11:14:02 STEP 1342 H= 335:30 +CPU= 57.628
  11:15:08 STEP 1343 H= 335:45 +CPU= 57.822
  11:16:33 STEP 1344 H= 336:00 +CPU= 74.588
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001344
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001344
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001248
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001260
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001272
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001284
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001296
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001308
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001320
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001332
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001332
Zipping up the intermediate file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_13.zip
Uploading the intermediate file: upload_file_13.zip
  11:17:34 STEP 1345 H= 336:15 +CPU= 54.352
  11:18:32 STEP 1346 H= 336:30 +CPU= 55.512
  11:19:32 STEP 1347 H= 336:45 +CPU= 56.144
  11:20:34 STEP 1348 H= 337:00 +CPU= 57.184
  11:21:34 STEP 1349 H= 337:15 +CPU= 54.470
  11:22:36 STEP 1350 H= 337:30 +CPU= 56.424
  11:23:37 STEP 1351 H= 337:45 +CPU= 56.053
  11:24:38 STEP 1352 H= 338:00 +CPU= 56.727
  11:25:39 STEP 1353 H= 338:15 +CPU= 54.750
  11:26:42 STEP 1354 H= 338:30 +CPU= 55.329
  11:27:46 STEP 1355 H= 338:45 +CPU= 57.373
  11:29:06 STEP 1356 H= 339:00 +CPU= 71.921
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001356
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001356
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001356
  11:30:09 STEP 1357 H= 339:15 +CPU= 56.734
  11:31:13 STEP 1358 H= 339:30 +CPU= 56.063
  11:32:17 STEP 1359 H= 339:45 +CPU= 56.990
  11:33:22 STEP 1360 H= 340:00 +CPU= 57.332
  11:34:27 STEP 1361 H= 340:15 +CPU= 56.400
  11:35:30 STEP 1362 H= 340:30 +CPU= 56.141
  11:36:34 STEP 1363 H= 340:45 +CPU= 56.629
  11:37:38 STEP 1364 H= 341:00 +CPU= 57.149
  11:38:42 STEP 1365 H= 341:15 +CPU= 57.606
  11:39:45 STEP 1366 H= 341:30 +CPU= 56.615
  11:40:49 STEP 1367 H= 341:45 +CPU= 56.048
  11:42:06 STEP 1368 H= 342:00 +CPU= 69.162
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001368
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001368
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001368
  11:43:06 STEP 1369 H= 342:15 +CPU= 55.974
  11:44:05 STEP 1370 H= 342:30 +CPU= 54.759
  11:45:06 STEP 1371 H= 342:45 +CPU= 54.992
  11:46:07 STEP 1372 H= 343:00 +CPU= 55.817
  11:47:03 STEP 1373 H= 343:15 +CPU= 54.907
  11:47:58 STEP 1374 H= 343:30 +CPU= 54.632
  11:48:55 STEP 1375 H= 343:45 +CPU= 55.910
  11:49:50 STEP 1376 H= 344:00 +CPU= 54.864
  11:50:45 STEP 1377 H= 344:15 +CPU= 54.462
  11:51:40 STEP 1378 H= 344:30 +CPU= 54.404
  11:52:36 STEP 1379 H= 344:45 +CPU= 56.069
  11:53:45 STEP 1380 H= 345:00 +CPU= 68.328
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001380
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001380
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001380
  11:54:40 STEP 1381 H= 345:15 +CPU= 54.154
  11:55:36 STEP 1382 H= 345:30 +CPU= 55.839
  11:56:32 STEP 1383 H= 345:45 +CPU= 54.674
  11:57:28 STEP 1384 H= 346:00 +CPU= 55.412
  11:58:23 STEP 1385 H= 346:15 +CPU= 55.462
  11:59:20 STEP 1386 H= 346:30 +CPU= 55.674
  12:00:16 STEP 1387 H= 346:45 +CPU= 55.552
  12:01:14 STEP 1388 H= 347:00 +CPU= 58.139
  12:02:09 STEP 1389 H= 347:15 +CPU= 53.841
  12:03:05 STEP 1390 H= 347:30 +CPU= 55.841
  12:04:02 STEP 1391 H= 347:45 +CPU= 56.322
  12:05:16 STEP 1392 H= 348:00 +CPU= 72.995
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001392
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001392
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001392
  12:06:10 STEP 1393 H= 348:15 +CPU= 53.573
  12:07:04 STEP 1394 H= 348:30 +CPU= 53.428
  12:07:58 STEP 1395 H= 348:45 +CPU= 53.200
  12:08:53 STEP 1396 H= 349:00 +CPU= 54.748
  12:09:48 STEP 1397 H= 349:15 +CPU= 54.085
  12:10:46 STEP 1398 H= 349:30 +CPU= 57.319
  12:11:41 STEP 1399 H= 349:45 +CPU= 54.451
  12:12:35 STEP 1400 H= 350:00 +CPU= 54.295
  12:13:32 STEP 1401 H= 350:15 +CPU= 55.967
  12:14:28 STEP 1402 H= 350:30 +CPU= 55.742
  12:15:22 STEP 1403 H= 350:45 +CPU= 53.142
  12:16:31 STEP 1404 H= 351:00 +CPU= 68.186
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001404
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001404
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001404
  12:17:24 STEP 1405 H= 351:15 +CPU= 52.638
  12:18:21 STEP 1406 H= 351:30 +CPU= 56.011
  12:19:15 STEP 1407 H= 351:45 +CPU= 53.542
  12:20:11 STEP 1408 H= 352:00 +CPU= 55.815
  12:21:05 STEP 1409 H= 352:15 +CPU= 53.651
  12:22:00 STEP 1410 H= 352:30 +CPU= 52.929
  12:22:56 STEP 1411 H= 352:45 +CPU= 55.408
  12:23:53 STEP 1412 H= 353:00 +CPU= 56.082
  12:24:50 STEP 1413 H= 353:15 +CPU= 56.319
  12:25:46 STEP 1414 H= 353:30 +CPU= 55.665
  12:26:41 STEP 1415 H= 353:45 +CPU= 55.031
  12:27:50 STEP 1416 H= 354:00 +CPU= 67.607
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001416
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001416
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001416
  12:28:46 STEP 1417 H= 354:15 +CPU= 56.212
  12:29:40 STEP 1418 H= 354:30 +CPU= 53.129
  12:30:35 STEP 1419 H= 354:45 +CPU= 54.653
  12:31:32 STEP 1420 H= 355:00 +CPU= 56.750
  12:32:27 STEP 1421 H= 355:15 +CPU= 54.533
  12:33:24 STEP 1422 H= 355:30 +CPU= 56.305
  12:34:21 STEP 1423 H= 355:45 +CPU= 56.090
  12:35:15 STEP 1424 H= 356:00 +CPU= 53.745
  12:36:11 STEP 1425 H= 356:15 +CPU= 55.164
  12:37:05 STEP 1426 H= 356:30 +CPU= 53.594
  12:38:00 STEP 1427 H= 356:45 +CPU= 54.877
  12:39:07 STEP 1428 H= 357:00 +CPU= 66.439
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001428
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001428
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001428
  12:40:02 STEP 1429 H= 357:15 +CPU= 53.796
  12:40:57 STEP 1430 H= 357:30 +CPU= 54.433
  12:41:52 STEP 1431 H= 357:45 +CPU= 54.656
  12:42:48 STEP 1432 H= 358:00 +CPU= 54.818
  12:43:43 STEP 1433 H= 358:15 +CPU= 54.403
  12:44:39 STEP 1434 H= 358:30 +CPU= 55.412
  12:45:34 STEP 1435 H= 358:45 +CPU= 54.592
  12:46:30 STEP 1436 H= 359:00 +CPU= 54.860
  12:47:25 STEP 1437 H= 359:15 +CPU= 54.647
  12:48:21 STEP 1438 H= 359:30 +CPU= 55.111
  12:49:17 STEP 1439 H= 359:45 +CPU= 54.571
  12:50:24 STEP 1440 H= 360:00 +CPU= 66.192
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.9017631849E-21      I       0.5432203650E-07      I
ICloud fractionI      0.0000000000E+00      I       0.2913048125E-02      I
I  Relat. Hum. I      0.4475881974E+00      I       0.4703568721E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.5040059131E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    318490436 BYTES IN      7 RECORDS TRANSFERRED IN    0.0009 SECONDS353878.3 Mbytes/s, TOTAL TIME=    0.0192( 4.7%)
 12:50:24 STEP 1440 H= 360:00 +CPU= 66.192
 END CNT3
 NSTEP =  1440 CNT0   000000000  48.863  48.863  49.689   0.983               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        43246/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    43246 ************   122.6     0.0   122.6     0.0   287.6
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.99      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                 1562   10500.0   10500.0     21.34      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    1960.5    1960.5      3.67      0.00
  10 SCAN2M - PHYSICS                             1441   22646.9   22646.9     42.46      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1655.2    1655.2      0.26      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63     116.6     116.6      0.01      0.00
  14 SUINIF                                          1    7459.7    7459.7      0.01      0.00
  17 GRIDFPOS IN CNT4                              121      61.8      61.8      0.01      0.00
  18 SUSPECG                                         1    3740.0    3740.0      0.00      0.00
  19 SUSPEC                                          1    3747.2    3747.2      0.00      0.00
  24 SUGRIDU                                         1    2911.0    2911.0      0.00      0.00
  25 SPECRT                                          1     367.4     367.4      0.00      0.00
  26 SUGRIDF                                         1     434.0     434.0      0.00      0.00
  27 RESTART FILES - WRITING                        30    6675.9    6675.9      0.26      0.00
  28 RESTART FILES - READING                         1       0.1       0.1      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                               121   11523.0   11523.0      1.81      0.00
  31 POSDDH IN STEPO                                 7       1.2       1.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441   10265.6   10265.6     19.25      0.00
  39 SU0YOMB                                         1     582.1     582.1      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      44.5      44.5      0.08      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440    1073.3    1073.3      2.01      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440     278.5     278.5      0.52      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.7       0.7      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     799.3     799.3      3.25      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     445.3     445.3      1.74      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     154.2     154.2      0.60      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     253.7     253.7      1.03      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      54.5      54.5      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125     140.6     140.6      0.57      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      78.6      78.6      0.31      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125     193.6     193.6      0.79      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      81.6      81.6      0.32      0.00
 400 GSTATS                                     167734       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                156066       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    76858.6 CPU TIME   75384.0 VECTOR TIME    75384.0


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY       16.9
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/steve/slots/16/ICMGGh7zg+001440
Moving to projects directory: /home/steve/slots/16/ICMSHh7zg+001440
Moving to projects directory: /home/steve/slots/16/ICMUAh7zg+001440
Adding to the zip: /home/steve/slots/16/NODE.001_01
Adding to the zip: /home/steve/slots/16/ifs.stat
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001404
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001392
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001380
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001344
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001368
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001416
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001440
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001428
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMUAh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMSHh7zg+001356
Adding to the zip: /home/steve/projects/climateprediction.net/oifs_43r3_bl_12285428/ICMGGh7zg+001404
Zipping up the final file: /home/steve/projects/climateprediction.net/oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
12:53:54 (919): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1_r1998477632_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 18:42:10 1536520 22439359 oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1 1,295,100 75,384 0.0582
08 Jun 2024 10:08:48 1536520 22439359 oifs_43r3_bl_a2s1_2016092300_20_1017_12285428_1 864,000 48,371 0.0560


©2024 climateprediction.net