climateprediction.net (CPDN) home page
Task 22314731

Task 22314731

Name oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1
Workunit 12209860
Created 23 Feb 2023, 21:59:13 UTC
Sent 23 Feb 2023, 22:00:18 UTC
Report deadline 24 Apr 2023, 22:00:18 UTC
Received 15 Apr 2023, 5:42:07 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1526341
Run time 12 hours 54 min 15 sec
CPU time 12 hours 45 min 29 sec
Validate state Valid
Credit 1,813.14
Device peak FLOPS 3.74 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.11
x86_64-pc-linux-gnu
Peak working set size 4.73 GB
Peak swap size 5.79 GB
Peak disk usage 1.41 GB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
5 +CPU= 29.407
  19:56:51 STEP  818 H= 204:30 +CPU= 28.305
  19:57:20 STEP  819 H= 204:45 +CPU= 29.002
  19:57:50 STEP  820 H= 205:00 +CPU= 29.410
  19:58:19 STEP  821 H= 205:15 +CPU= 29.014
  19:58:48 STEP  822 H= 205:30 +CPU= 28.691
  19:59:17 STEP  823 H= 205:45 +CPU= 29.614
  19:59:47 STEP  824 H= 206:00 +CPU= 29.650
  20:00:17 STEP  825 H= 206:15 +CPU= 29.232
  20:00:47 STEP  826 H= 206:30 +CPU= 29.762
  20:01:16 STEP  827 H= 206:45 +CPU= 29.475
  20:01:53 STEP  828 H= 207:00 +CPU= 36.892
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000828
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000828
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000828
  20:02:22 STEP  829 H= 207:15 +CPU= 28.843
  20:02:51 STEP  830 H= 207:30 +CPU= 28.624
  20:03:22 STEP  831 H= 207:45 +CPU= 30.415
  20:03:52 STEP  832 H= 208:00 +CPU= 30.611
  20:04:22 STEP  833 H= 208:15 +CPU= 29.491
  20:04:52 STEP  834 H= 208:30 +CPU= 30.212
  20:05:23 STEP  835 H= 208:45 +CPU= 29.928
  20:05:53 STEP  836 H= 209:00 +CPU= 29.716
  20:06:22 STEP  837 H= 209:15 +CPU= 29.113
  20:06:52 STEP  838 H= 209:30 +CPU= 30.043
  20:07:23 STEP  839 H= 209:45 +CPU= 31.262
  20:08:01 STEP  840 H= 210:00 +CPU= 37.047
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000840
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000840
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000840
  20:08:30 STEP  841 H= 210:15 +CPU= 29.078
  20:09:00 STEP  842 H= 210:30 +CPU= 29.423
  20:09:30 STEP  843 H= 210:45 +CPU= 29.779
  20:09:59 STEP  844 H= 211:00 +CPU= 29.443
  20:10:30 STEP  845 H= 211:15 +CPU= 30.269
  20:10:59 STEP  846 H= 211:30 +CPU= 29.019
  20:11:30 STEP  847 H= 211:45 +CPU= 30.738
  20:12:00 STEP  848 H= 212:00 +CPU= 30.480
  20:12:29 STEP  849 H= 212:15 +CPU= 28.750
  20:13:00 STEP  850 H= 212:30 +CPU= 30.254
  20:13:30 STEP  851 H= 212:45 +CPU= 29.714
  20:14:07 STEP  852 H= 213:00 +CPU= 37.580
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000852
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000852
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000852
  20:14:38 STEP  853 H= 213:15 +CPU= 30.038
  20:15:07 STEP  854 H= 213:30 +CPU= 28.728
  20:15:36 STEP  855 H= 213:45 +CPU= 29.374
  20:16:06 STEP  856 H= 214:00 +CPU= 30.135
  20:16:36 STEP  857 H= 214:15 +CPU= 29.267
  20:17:06 STEP  858 H= 214:30 +CPU= 30.086
  20:17:36 STEP  859 H= 214:45 +CPU= 29.625
  20:18:05 STEP  860 H= 215:00 +CPU= 28.769
  20:18:34 STEP  861 H= 215:15 +CPU= 29.004
  20:19:04 STEP  862 H= 215:30 +CPU= 29.307
  20:19:33 STEP  863 H= 215:45 +CPU= 28.434
  20:20:13 STEP  864 H= 216:00 +CPU= 40.127
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000864
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000864
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000864
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000768
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000768
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000768
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000780
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000780
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000780
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000792
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000792
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000792
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000804
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000804
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000804
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000816
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000816
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000816
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000828
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000828
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000828
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000840
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000840
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000840
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000852
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000852
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000852
Zipping up the intermediate file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_8.zip
Uploading the intermediate file: upload_file_8.zip
  20:20:43 STEP  865 H= 216:15 +CPU= 30.109
  20:21:14 STEP  866 H= 216:30 +CPU= 30.516
  20:21:44 STEP  867 H= 216:45 +CPU= 29.975
  20:22:17 STEP  868 H= 217:00 +CPU= 32.332
  20:22:47 STEP  869 H= 217:15 +CPU= 30.511
  20:23:19 STEP  870 H= 217:30 +CPU= 31.857
  20:23:50 STEP  871 H= 217:45 +CPU= 30.411
  20:24:21 STEP  872 H= 218:00 +CPU= 31.469
  20:24:52 STEP  873 H= 218:15 +CPU= 30.544
  20:25:23 STEP  874 H= 218:30 +CPU= 30.614
  20:25:53 STEP  875 H= 218:45 +CPU= 29.923
  20:26:31 STEP  876 H= 219:00 +CPU= 38.521
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000876
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000876
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000876
  20:27:03 STEP  877 H= 219:15 +CPU= 31.021
  20:27:33 STEP  878 H= 219:30 +CPU= 30.545
  20:28:04 STEP  879 H= 219:45 +CPU= 30.761
  20:28:36 STEP  880 H= 220:00 +CPU= 31.313
  20:29:07 STEP  881 H= 220:15 +CPU= 31.117
  20:29:38 STEP  882 H= 220:30 +CPU= 31.055
  20:30:08 STEP  883 H= 220:45 +CPU= 29.709
  20:30:40 STEP  884 H= 221:00 +CPU= 31.594
  20:31:10 STEP  885 H= 221:15 +CPU= 29.699
  20:31:40 STEP  886 H= 221:30 +CPU= 30.356
  20:32:11 STEP  887 H= 221:45 +CPU= 30.497
  20:32:50 STEP  888 H= 222:00 +CPU= 38.759
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000888
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000888
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000888
  20:33:20 STEP  889 H= 222:15 +CPU= 29.741
  20:33:50 STEP  890 H= 222:30 +CPU= 30.635
  20:34:21 STEP  891 H= 222:45 +CPU= 30.795
  20:34:53 STEP  892 H= 223:00 +CPU= 31.145
  20:35:24 STEP  893 H= 223:15 +CPU= 31.111
  20:35:55 STEP  894 H= 223:30 +CPU= 30.945
  20:36:26 STEP  895 H= 223:45 +CPU= 30.786
  20:36:58 STEP  896 H= 224:00 +CPU= 32.298
  20:37:28 STEP  897 H= 224:15 +CPU= 29.933
  20:37:59 STEP  898 H= 224:30 +CPU= 30.543
  20:38:29 STEP  899 H= 224:45 +CPU= 29.982
  20:39:08 STEP  900 H= 225:00 +CPU= 38.698
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000900
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000900
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000900
  20:39:40 STEP  901 H= 225:15 +CPU= 31.525
  20:40:10 STEP  902 H= 225:30 +CPU= 30.227
  20:40:40 STEP  903 H= 225:45 +CPU= 30.116
  20:41:11 STEP  904 H= 226:00 +CPU= 30.740
  20:41:41 STEP  905 H= 226:15 +CPU= 29.739
  20:42:11 STEP  906 H= 226:30 +CPU= 29.943
  20:42:43 STEP  907 H= 226:45 +CPU= 31.462
  20:43:12 STEP  908 H= 227:00 +CPU= 29.464
  20:43:42 STEP  909 H= 227:15 +CPU= 29.883
  20:44:12 STEP  910 H= 227:30 +CPU= 29.833
  20:44:43 STEP  911 H= 227:45 +CPU= 30.282
  20:45:27 STEP  912 H= 228:00 +CPU= 44.518
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000912
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000912
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000912
  20:46:00 STEP  913 H= 228:15 +CPU= 31.948
  20:46:31 STEP  914 H= 228:30 +CPU= 31.463
  20:47:03 STEP  915 H= 228:45 +CPU= 31.474
  20:47:34 STEP  916 H= 229:00 +CPU= 31.090
  20:48:05 STEP  917 H= 229:15 +CPU= 30.776
  20:48:36 STEP  918 H= 229:30 +CPU= 30.490
  20:49:06 STEP  919 H= 229:45 +CPU= 30.485
  20:49:37 STEP  920 H= 230:00 +CPU= 30.996
  20:50:08 STEP  921 H= 230:15 +CPU= 31.033
  20:50:41 STEP  922 H= 230:30 +CPU= 32.177
  20:51:12 STEP  923 H= 230:45 +CPU= 31.011
  20:51:51 STEP  924 H= 231:00 +CPU= 38.467
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000924
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000924
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000924
  20:52:21 STEP  925 H= 231:15 +CPU= 29.996
  20:52:51 STEP  926 H= 231:30 +CPU= 30.533
  20:53:23 STEP  927 H= 231:45 +CPU= 31.777
  20:53:55 STEP  928 H= 232:00 +CPU= 31.411
  20:54:26 STEP  929 H= 232:15 +CPU= 30.999
  20:54:59 STEP  930 H= 232:30 +CPU= 32.573
  20:55:28 STEP  931 H= 232:45 +CPU= 29.550
  20:56:00 STEP  932 H= 233:00 +CPU= 31.193
  20:56:30 STEP  933 H= 233:15 +CPU= 30.604
  20:57:01 STEP  934 H= 233:30 +CPU= 29.924
  20:57:32 STEP  935 H= 233:45 +CPU= 31.451
  20:58:10 STEP  936 H= 234:00 +CPU= 38.140
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000936
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000936
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000936
  20:58:41 STEP  937 H= 234:15 +CPU= 30.130
  20:59:11 STEP  938 H= 234:30 +CPU= 30.667
  20:59:41 STEP  939 H= 234:45 +CPU= 29.418
  21:00:12 STEP  940 H= 235:00 +CPU= 31.096
  21:00:44 STEP  941 H= 235:15 +CPU= 31.387
  21:01:15 STEP  942 H= 235:30 +CPU= 31.477
  21:01:46 STEP  943 H= 235:45 +CPU= 30.193
  21:02:18 STEP  944 H= 236:00 +CPU= 31.718
  21:02:48 STEP  945 H= 236:15 +CPU= 30.775
  21:03:19 STEP  946 H= 236:30 +CPU= 30.786
  21:03:51 STEP  947 H= 236:45 +CPU= 31.087
  21:04:31 STEP  948 H= 237:00 +CPU= 40.614
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000948
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000948
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000948
  21:05:02 STEP  949 H= 237:15 +CPU= 30.298
  21:05:33 STEP  950 H= 237:30 +CPU= 30.742
  21:06:04 STEP  951 H= 237:45 +CPU= 31.305
  21:06:36 STEP  952 H= 238:00 +CPU= 31.614
  21:07:07 STEP  953 H= 238:15 +CPU= 31.020
  21:07:38 STEP  954 H= 238:30 +CPU= 30.529
  21:08:08 STEP  955 H= 238:45 +CPU= 30.081
  21:08:39 STEP  956 H= 239:00 +CPU= 30.764
  21:09:11 STEP  957 H= 239:15 +CPU= 32.117
  21:09:42 STEP  958 H= 239:30 +CPU= 30.973
  21:10:14 STEP  959 H= 239:45 +CPU= 31.667
  21:10:59 STEP  960 H= 240:00 +CPU= 44.332
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000960
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000960
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000960
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000864
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000864
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000864
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000876
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000876
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000876
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000888
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000888
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000888
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000900
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000900
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000900
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000912
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000912
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000912
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000924
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000924
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000924
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000936
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000936
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000936
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000948
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000948
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000948
Zipping up the intermediate file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_9.zip
Uploading the intermediate file: upload_file_9.zip
  21:11:30 STEP  961 H= 240:15 +CPU= 30.772
Uploading trickle at timestep: 864000
  21:12:00 STEP  962 H= 240:30 +CPU= 30.725
  21:12:31 STEP  963 H= 240:45 +CPU= 30.403
  21:13:03 STEP  964 H= 241:00 +CPU= 31.607
  21:13:34 STEP  965 H= 241:15 +CPU= 30.725
  21:14:05 STEP  966 H= 241:30 +CPU= 31.157
  21:14:35 STEP  967 H= 241:45 +CPU= 29.884
  21:15:07 STEP  968 H= 242:00 +CPU= 32.057
  21:15:39 STEP  969 H= 242:15 +CPU= 31.700
  21:16:11 STEP  970 H= 242:30 +CPU= 31.604
  21:16:42 STEP  971 H= 242:45 +CPU= 31.336
  21:17:20 STEP  972 H= 243:00 +CPU= 37.505
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000972
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000972
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000972
  21:17:50 STEP  973 H= 243:15 +CPU= 30.058
  21:18:21 STEP  974 H= 243:30 +CPU= 30.503
  21:18:51 STEP  975 H= 243:45 +CPU= 30.385
  21:19:23 STEP  976 H= 244:00 +CPU= 31.791
  21:19:53 STEP  977 H= 244:15 +CPU= 30.253
  21:20:25 STEP  978 H= 244:30 +CPU= 31.150
  21:20:56 STEP  979 H= 244:45 +CPU= 31.320
  21:21:28 STEP  980 H= 245:00 +CPU= 31.243
  21:21:58 STEP  981 H= 245:15 +CPU= 30.691
  21:22:30 STEP  982 H= 245:30 +CPU= 31.432
  21:23:02 STEP  983 H= 245:45 +CPU= 31.522
  21:23:40 STEP  984 H= 246:00 +CPU= 38.270
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000984
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000984
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000984
  21:24:11 STEP  985 H= 246:15 +CPU= 30.948
  21:24:42 STEP  986 H= 246:30 +CPU= 31.107
  21:25:14 STEP  987 H= 246:45 +CPU= 31.465
  21:25:46 STEP  988 H= 247:00 +CPU= 31.504
  21:26:16 STEP  989 H= 247:15 +CPU= 30.437
  21:26:46 STEP  990 H= 247:30 +CPU= 30.150
  21:27:18 STEP  991 H= 247:45 +CPU= 30.948
  21:27:50 STEP  992 H= 248:00 +CPU= 32.128
  21:28:21 STEP  993 H= 248:15 +CPU= 31.175
  21:28:53 STEP  994 H= 248:30 +CPU= 31.500
  21:29:23 STEP  995 H= 248:45 +CPU= 30.251
  21:30:02 STEP  996 H= 249:00 +CPU= 38.915
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+000996
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+000996
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+000996
  21:30:33 STEP  997 H= 249:15 +CPU= 30.802
  21:31:05 STEP  998 H= 249:30 +CPU= 31.155
  21:31:35 STEP  999 H= 249:45 +CPU= 29.834
  21:32:07 STEP 1000 H= 250:00 +CPU= 32.057
  21:32:37 STEP 1001 H= 250:15 +CPU= 30.353
  21:33:10 STEP 1002 H= 250:30 +CPU= 32.097
  21:33:41 STEP 1003 H= 250:45 +CPU= 30.954
  21:34:12 STEP 1004 H= 251:00 +CPU= 31.366
  21:34:43 STEP 1005 H= 251:15 +CPU= 30.663
  21:35:13 STEP 1006 H= 251:30 +CPU= 29.926
  21:35:44 STEP 1007 H= 251:45 +CPU= 31.151
  21:36:28 STEP 1008 H= 252:00 +CPU= 43.194
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001008
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001008
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001008
  21:36:58 STEP 1009 H= 252:15 +CPU= 29.986
  21:37:29 STEP 1010 H= 252:30 +CPU= 30.942
  21:38:00 STEP 1011 H= 252:45 +CPU= 31.104
  21:38:33 STEP 1012 H= 253:00 +CPU= 32.488
  21:39:04 STEP 1013 H= 253:15 +CPU= 30.955
  21:39:34 STEP 1014 H= 253:30 +CPU= 29.864
  21:40:05 STEP 1015 H= 253:45 +CPU= 30.556
  21:40:37 STEP 1016 H= 254:00 +CPU= 32.022
  21:41:07 STEP 1017 H= 254:15 +CPU= 30.110
  21:41:36 STEP 1018 H= 254:30 +CPU= 29.286
  21:42:07 STEP 1019 H= 254:45 +CPU= 30.573
  21:42:46 STEP 1020 H= 255:00 +CPU= 39.081
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001020
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001020
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001020
  21:43:17 STEP 1021 H= 255:15 +CPU= 30.891
  21:43:48 STEP 1022 H= 255:30 +CPU= 30.863
  21:44:20 STEP 1023 H= 255:45 +CPU= 31.321
  21:44:51 STEP 1024 H= 256:00 +CPU= 30.725
  21:45:22 STEP 1025 H= 256:15 +CPU= 30.746
  21:45:52 STEP 1026 H= 256:30 +CPU= 30.399
  21:46:23 STEP 1027 H= 256:45 +CPU= 30.378
  21:46:55 STEP 1028 H= 257:00 +CPU= 32.066
  21:47:26 STEP 1029 H= 257:15 +CPU= 30.919
  21:47:56 STEP 1030 H= 257:30 +CPU= 30.173
  21:48:27 STEP 1031 H= 257:45 +CPU= 30.526
  21:49:06 STEP 1032 H= 258:00 +CPU= 38.702
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001032
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001032
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001032
  21:49:36 STEP 1033 H= 258:15 +CPU= 29.844
  21:50:07 STEP 1034 H= 258:30 +CPU= 30.783
  21:50:37 STEP 1035 H= 258:45 +CPU= 30.419
  21:51:09 STEP 1036 H= 259:00 +CPU= 31.781
  21:51:40 STEP 1037 H= 259:15 +CPU= 30.964
  21:52:11 STEP 1038 H= 259:30 +CPU= 30.876
  21:52:42 STEP 1039 H= 259:45 +CPU= 30.284
  21:53:12 STEP 1040 H= 260:00 +CPU= 30.455
  21:53:43 STEP 1041 H= 260:15 +CPU= 30.359
  21:54:14 STEP 1042 H= 260:30 +CPU= 30.477
  21:54:45 STEP 1043 H= 260:45 +CPU= 31.282
  21:55:25 STEP 1044 H= 261:00 +CPU= 39.538
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001044
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001044
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001044
  21:55:55 STEP 1045 H= 261:15 +CPU= 30.246
  21:56:26 STEP 1046 H= 261:30 +CPU= 30.883
  21:56:57 STEP 1047 H= 261:45 +CPU= 30.900
  21:57:30 STEP 1048 H= 262:00 +CPU= 32.533
  21:58:01 STEP 1049 H= 262:15 +CPU= 31.186
  21:58:32 STEP 1050 H= 262:30 +CPU= 30.479
  21:59:03 STEP 1051 H= 262:45 +CPU= 30.977
  21:59:34 STEP 1052 H= 263:00 +CPU= 31.607
  22:00:05 STEP 1053 H= 263:15 +CPU= 30.293
  22:00:36 STEP 1054 H= 263:30 +CPU= 31.154
  22:01:07 STEP 1055 H= 263:45 +CPU= 30.990
  22:01:52 STEP 1056 H= 264:00 +CPU= 44.429
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001056
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001056
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001056
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000960
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000960
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000960
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000972
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000972
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000972
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000984
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000984
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000984
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+000996
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+000996
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+000996
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001008
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001008
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001008
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001020
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001020
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001020
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001032
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001032
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001032
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001044
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001044
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001044
Zipping up the intermediate file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_10.zip
Uploading the intermediate file: upload_file_10.zip
  22:02:23 STEP 1057 H= 264:15 +CPU= 30.646
  22:02:55 STEP 1058 H= 264:30 +CPU= 31.850
  22:03:27 STEP 1059 H= 264:45 +CPU= 32.176
  22:04:00 STEP 1060 H= 265:00 +CPU= 32.402
  22:04:31 STEP 1061 H= 265:15 +CPU= 30.946
  22:05:02 STEP 1062 H= 265:30 +CPU= 31.224
  22:05:33 STEP 1063 H= 265:45 +CPU= 30.541
  22:06:06 STEP 1064 H= 266:00 +CPU= 32.453
  22:06:36 STEP 1065 H= 266:15 +CPU= 30.765
  22:07:07 STEP 1066 H= 266:30 +CPU= 30.013
  22:07:37 STEP 1067 H= 266:45 +CPU= 30.475
  22:08:16 STEP 1068 H= 267:00 +CPU= 38.451
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001068
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001068
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001068
  22:08:46 STEP 1069 H= 267:15 +CPU= 30.497
  22:09:17 STEP 1070 H= 267:30 +CPU= 30.607
  22:09:52 STEP 1071 H= 267:45 +CPU= 32.055
  22:10:25 STEP 1072 H= 268:00 +CPU= 32.824
  22:10:56 STEP 1073 H= 268:15 +CPU= 30.560
  22:11:27 STEP 1074 H= 268:30 +CPU= 30.972
  22:11:59 STEP 1075 H= 268:45 +CPU= 31.689
  22:12:30 STEP 1076 H= 269:00 +CPU= 30.426
  22:13:00 STEP 1077 H= 269:15 +CPU= 30.221
  22:13:31 STEP 1078 H= 269:30 +CPU= 30.731
  22:14:02 STEP 1079 H= 269:45 +CPU= 31.136
  22:14:42 STEP 1080 H= 270:00 +CPU= 39.644
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001080
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001080
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001080
  22:15:13 STEP 1081 H= 270:15 +CPU= 31.493
  22:15:44 STEP 1082 H= 270:30 +CPU= 30.840
  22:16:16 STEP 1083 H= 270:45 +CPU= 31.288
  22:16:48 STEP 1084 H= 271:00 +CPU= 31.625
  22:17:19 STEP 1085 H= 271:15 +CPU= 31.103
  22:17:50 STEP 1086 H= 271:30 +CPU= 30.812
  22:18:22 STEP 1087 H= 271:45 +CPU= 31.975
  22:18:54 STEP 1088 H= 272:00 +CPU= 31.591
  22:19:25 STEP 1089 H= 272:15 +CPU= 31.024
  22:19:57 STEP 1090 H= 272:30 +CPU= 31.493
  22:20:27 STEP 1091 H= 272:45 +CPU= 30.791
  22:21:06 STEP 1092 H= 273:00 +CPU= 38.566
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001092
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001092
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001092
  22:21:37 STEP 1093 H= 273:15 +CPU= 30.795
  22:22:08 STEP 1094 H= 273:30 +CPU= 30.209
  22:22:39 STEP 1095 H= 273:45 +CPU= 31.162
  22:23:10 STEP 1096 H= 274:00 +CPU= 31.191
  22:23:41 STEP 1097 H= 274:15 +CPU= 30.765
  22:24:13 STEP 1098 H= 274:30 +CPU= 30.985
  22:24:43 STEP 1099 H= 274:45 +CPU= 30.218
  22:25:14 STEP 1100 H= 275:00 +CPU= 30.880
  22:25:45 STEP 1101 H= 275:15 +CPU= 30.364
  22:26:15 STEP 1102 H= 275:30 +CPU= 30.163
  22:26:46 STEP 1103 H= 275:45 +CPU= 30.669
  22:27:29 STEP 1104 H= 276:00 +CPU= 42.976
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001104
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001104
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001104
  22:28:00 STEP 1105 H= 276:15 +CPU= 30.226
  22:28:31 STEP 1106 H= 276:30 +CPU= 31.545
  22:29:02 STEP 1107 H= 276:45 +CPU= 30.814
  22:29:34 STEP 1108 H= 277:00 +CPU= 31.467
  22:30:04 STEP 1109 H= 277:15 +CPU= 29.833
  22:30:35 STEP 1110 H= 277:30 +CPU= 30.609
  22:31:05 STEP 1111 H= 277:45 +CPU= 30.097
  22:31:36 STEP 1112 H= 278:00 +CPU= 31.032
  22:32:07 STEP 1113 H= 278:15 +CPU= 30.799
  22:32:38 STEP 1114 H= 278:30 +CPU= 30.662
  22:33:09 STEP 1115 H= 278:45 +CPU= 30.661
  22:33:49 STEP 1116 H= 279:00 +CPU= 40.258
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001116
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001116
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001116
  22:34:21 STEP 1117 H= 279:15 +CPU= 31.419
  22:34:52 STEP 1118 H= 279:30 +CPU= 30.551
  22:35:23 STEP 1119 H= 279:45 +CPU= 31.276
  22:35:56 STEP 1120 H= 280:00 +CPU= 33.124
  22:36:28 STEP 1121 H= 280:15 +CPU= 31.132
  22:36:59 STEP 1122 H= 280:30 +CPU= 30.931
  22:37:30 STEP 1123 H= 280:45 +CPU= 30.963
  22:38:02 STEP 1124 H= 281:00 +CPU= 31.963
  22:38:33 STEP 1125 H= 281:15 +CPU= 31.179
  22:39:04 STEP 1126 H= 281:30 +CPU= 30.925
  22:39:34 STEP 1127 H= 281:45 +CPU= 29.992
  22:40:13 STEP 1128 H= 282:00 +CPU= 37.940
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001128
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001128
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001128
  22:40:44 STEP 1129 H= 282:15 +CPU= 31.277
  22:41:15 STEP 1130 H= 282:30 +CPU= 30.864
  22:41:45 STEP 1131 H= 282:45 +CPU= 30.307
  22:42:16 STEP 1132 H= 283:00 +CPU= 30.804
  22:42:49 STEP 1133 H= 283:15 +CPU= 32.320
  22:43:20 STEP 1134 H= 283:30 +CPU= 31.287
  22:43:51 STEP 1135 H= 283:45 +CPU= 30.662
  22:44:23 STEP 1136 H= 284:00 +CPU= 31.919
  22:44:54 STEP 1137 H= 284:15 +CPU= 31.244
  22:45:25 STEP 1138 H= 284:30 +CPU= 30.855
  22:45:57 STEP 1139 H= 284:45 +CPU= 31.492
  22:46:37 STEP 1140 H= 285:00 +CPU= 40.067
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001140
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001140
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001140
  22:47:10 STEP 1141 H= 285:15 +CPU= 32.030
  22:47:40 STEP 1142 H= 285:30 +CPU= 30.321
  22:48:10 STEP 1143 H= 285:45 +CPU= 29.514
  22:48:42 STEP 1144 H= 286:00 +CPU= 32.579
  22:49:13 STEP 1145 H= 286:15 +CPU= 30.243
  22:49:44 STEP 1146 H= 286:30 +CPU= 31.256
  22:50:16 STEP 1147 H= 286:45 +CPU= 31.364
  22:50:47 STEP 1148 H= 287:00 +CPU= 31.193
  22:51:18 STEP 1149 H= 287:15 +CPU= 30.574
  22:51:48 STEP 1150 H= 287:30 +CPU= 30.270
  22:52:20 STEP 1151 H= 287:45 +CPU= 32.175
  22:53:03 STEP 1152 H= 288:00 +CPU= 42.878
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001152
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001152
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001152
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001056
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001056
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001056
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001068
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001068
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001068
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001080
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001080
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001080
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001092
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001092
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001092
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001104
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001104
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001104
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001116
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001116
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001116
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001128
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001128
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001128
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001140
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001140
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001140
Zipping up the intermediate file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_11.zip
Uploading the intermediate file: upload_file_11.zip
  22:53:34 STEP 1153 H= 288:15 +CPU= 30.546
  22:54:06 STEP 1154 H= 288:30 +CPU= 31.738
  22:54:37 STEP 1155 H= 288:45 +CPU= 30.579
  22:55:09 STEP 1156 H= 289:00 +CPU= 31.673
  22:55:39 STEP 1157 H= 289:15 +CPU= 30.685
  22:56:10 STEP 1158 H= 289:30 +CPU= 30.694
  22:56:42 STEP 1159 H= 289:45 +CPU= 31.801
  22:57:14 STEP 1160 H= 290:00 +CPU= 32.003
  22:57:46 STEP 1161 H= 290:15 +CPU= 31.449
  22:58:17 STEP 1162 H= 290:30 +CPU= 30.872
  22:58:49 STEP 1163 H= 290:45 +CPU= 31.349
  22:59:27 STEP 1164 H= 291:00 +CPU= 38.503
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001164
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001164
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001164
  22:59:58 STEP 1165 H= 291:15 +CPU= 30.482
  23:00:30 STEP 1166 H= 291:30 +CPU= 31.851
  23:01:01 STEP 1167 H= 291:45 +CPU= 30.923
  23:01:34 STEP 1168 H= 292:00 +CPU= 32.134
  23:02:05 STEP 1169 H= 292:15 +CPU= 30.905
  23:02:37 STEP 1170 H= 292:30 +CPU= 32.277
  23:03:08 STEP 1171 H= 292:45 +CPU= 30.590
  23:03:39 STEP 1172 H= 293:00 +CPU= 31.389
  23:04:10 STEP 1173 H= 293:15 +CPU= 30.508
  23:04:41 STEP 1174 H= 293:30 +CPU= 30.792
  23:05:12 STEP 1175 H= 293:45 +CPU= 30.941
  23:05:52 STEP 1176 H= 294:00 +CPU= 39.818
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001176
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001176
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001176
  23:06:23 STEP 1177 H= 294:15 +CPU= 30.082
  23:06:54 STEP 1178 H= 294:30 +CPU= 31.263
  23:07:26 STEP 1179 H= 294:45 +CPU= 31.651
  23:07:59 STEP 1180 H= 295:00 +CPU= 32.638
  23:08:30 STEP 1181 H= 295:15 +CPU= 30.849
  23:09:01 STEP 1182 H= 295:30 +CPU= 31.392
  23:09:32 STEP 1183 H= 295:45 +CPU= 30.447
  23:10:03 STEP 1184 H= 296:00 +CPU= 31.299
  23:10:34 STEP 1185 H= 296:15 +CPU= 30.442
  23:11:05 STEP 1186 H= 296:30 +CPU= 30.982
  23:11:37 STEP 1187 H= 296:45 +CPU= 31.497
  23:12:15 STEP 1188 H= 297:00 +CPU= 38.695
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001188
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001188
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001188
  23:12:46 STEP 1189 H= 297:15 +CPU= 30.734
  23:13:18 STEP 1190 H= 297:30 +CPU= 31.652
  23:13:50 STEP 1191 H= 297:45 +CPU= 31.414
  23:14:22 STEP 1192 H= 298:00 +CPU= 31.800
  23:14:53 STEP 1193 H= 298:15 +CPU= 30.734
  23:15:22 STEP 1194 H= 298:30 +CPU= 29.796
  23:15:52 STEP 1195 H= 298:45 +CPU= 29.822
  23:16:24 STEP 1196 H= 299:00 +CPU= 31.847
  23:16:55 STEP 1197 H= 299:15 +CPU= 30.498
  23:17:26 STEP 1198 H= 299:30 +CPU= 30.880
  23:17:58 STEP 1199 H= 299:45 +CPU= 31.496
  23:18:43 STEP 1200 H= 300:00 +CPU= 44.938
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001200
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001200
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001200
  23:19:13 STEP 1201 H= 300:15 +CPU= 30.609
  23:19:44 STEP 1202 H= 300:30 +CPU= 30.483
  23:20:14 STEP 1203 H= 300:45 +CPU= 29.874
  23:20:46 STEP 1204 H= 301:00 +CPU= 31.764
  23:21:17 STEP 1205 H= 301:15 +CPU= 30.838
  23:21:49 STEP 1206 H= 301:30 +CPU= 31.526
  23:22:19 STEP 1207 H= 301:45 +CPU= 30.042
  23:22:50 STEP 1208 H= 302:00 +CPU= 31.144
  23:23:20 STEP 1209 H= 302:15 +CPU= 29.901
  23:23:51 STEP 1210 H= 302:30 +CPU= 31.014
  23:24:22 STEP 1211 H= 302:45 +CPU= 30.840
  23:25:01 STEP 1212 H= 303:00 +CPU= 38.347
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001212
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001212
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001212
  23:25:31 STEP 1213 H= 303:15 +CPU= 30.315
  23:26:03 STEP 1214 H= 303:30 +CPU= 31.197
  23:26:34 STEP 1215 H= 303:45 +CPU= 31.014
  23:27:05 STEP 1216 H= 304:00 +CPU= 31.202
  23:27:36 STEP 1217 H= 304:15 +CPU= 31.254
  23:28:07 STEP 1218 H= 304:30 +CPU= 30.840
  23:28:39 STEP 1219 H= 304:45 +CPU= 31.648
  23:29:10 STEP 1220 H= 305:00 +CPU= 30.704
  23:29:41 STEP 1221 H= 305:15 +CPU= 30.538
  23:30:12 STEP 1222 H= 305:30 +CPU= 30.701
  23:30:42 STEP 1223 H= 305:45 +CPU= 30.302
  23:31:20 STEP 1224 H= 306:00 +CPU= 37.407
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001224
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001224
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001224
  23:31:52 STEP 1225 H= 306:15 +CPU= 31.789
  23:32:22 STEP 1226 H= 306:30 +CPU= 30.787
  23:32:54 STEP 1227 H= 306:45 +CPU= 31.503
  23:33:25 STEP 1228 H= 307:00 +CPU= 30.856
  23:33:56 STEP 1229 H= 307:15 +CPU= 31.208
  23:34:27 STEP 1230 H= 307:30 +CPU= 30.821
  23:34:58 STEP 1231 H= 307:45 +CPU= 30.204
  23:35:29 STEP 1232 H= 308:00 +CPU= 30.972
  23:36:00 STEP 1233 H= 308:15 +CPU= 31.235
  23:36:30 STEP 1234 H= 308:30 +CPU= 30.108
  23:37:03 STEP 1235 H= 308:45 +CPU= 32.547
  23:37:42 STEP 1236 H= 309:00 +CPU= 38.277
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001236
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001236
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001236
  23:38:12 STEP 1237 H= 309:15 +CPU= 30.162
  23:38:42 STEP 1238 H= 309:30 +CPU= 29.928
  23:39:13 STEP 1239 H= 309:45 +CPU= 30.367
  23:39:43 STEP 1240 H= 310:00 +CPU= 30.615
  23:40:15 STEP 1241 H= 310:15 +CPU= 31.544
  23:40:46 STEP 1242 H= 310:30 +CPU= 30.864
  23:41:17 STEP 1243 H= 310:45 +CPU= 31.169
  23:41:48 STEP 1244 H= 311:00 +CPU= 30.839
  23:42:19 STEP 1245 H= 311:15 +CPU= 30.565
  23:42:50 STEP 1246 H= 311:30 +CPU= 31.196
  23:43:23 STEP 1247 H= 311:45 +CPU= 33.113
  23:44:08 STEP 1248 H= 312:00 +CPU= 43.931
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001248
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001248
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001248
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001152
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001152
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001152
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001164
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001164
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001164
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001176
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001176
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001176
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001188
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001188
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001188
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001200
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001200
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001200
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001212
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001212
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001212
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001224
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001224
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001224
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001236
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001236
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001236
Zipping up the intermediate file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_12.zip
Uploading the intermediate file: upload_file_12.zip
  23:44:39 STEP 1249 H= 312:15 +CPU= 31.724
  23:45:10 STEP 1250 H= 312:30 +CPU= 30.748
  23:45:42 STEP 1251 H= 312:45 +CPU= 31.133
  23:46:13 STEP 1252 H= 313:00 +CPU= 31.626
  23:46:44 STEP 1253 H= 313:15 +CPU= 30.431
  23:47:15 STEP 1254 H= 313:30 +CPU= 30.506
  23:47:45 STEP 1255 H= 313:45 +CPU= 30.628
  23:48:18 STEP 1256 H= 314:00 +CPU= 32.028
  23:48:49 STEP 1257 H= 314:15 +CPU= 30.921
  23:49:20 STEP 1258 H= 314:30 +CPU= 31.179
  23:49:52 STEP 1259 H= 314:45 +CPU= 31.819
  23:50:31 STEP 1260 H= 315:00 +CPU= 39.215
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001260
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001260
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001260
  23:51:03 STEP 1261 H= 315:15 +CPU= 31.135
  23:51:34 STEP 1262 H= 315:30 +CPU= 30.831
  23:52:05 STEP 1263 H= 315:45 +CPU= 31.507
  23:52:37 STEP 1264 H= 316:00 +CPU= 31.934
  23:53:09 STEP 1265 H= 316:15 +CPU= 31.884
  23:53:40 STEP 1266 H= 316:30 +CPU= 30.661
  23:54:11 STEP 1267 H= 316:45 +CPU= 31.114
  23:54:42 STEP 1268 H= 317:00 +CPU= 30.885
  23:55:13 STEP 1269 H= 317:15 +CPU= 30.728
  23:55:44 STEP 1270 H= 317:30 +CPU= 30.722
  23:56:16 STEP 1271 H= 317:45 +CPU= 31.407
  23:56:55 STEP 1272 H= 318:00 +CPU= 39.389
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001272
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001272
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001272
  23:57:27 STEP 1273 H= 318:15 +CPU= 31.582
  23:57:59 STEP 1274 H= 318:30 +CPU= 31.644
  23:58:29 STEP 1275 H= 318:45 +CPU= 30.080
  23:59:02 STEP 1276 H= 319:00 +CPU= 32.073
  23:59:33 STEP 1277 H= 319:15 +CPU= 31.367
  00:00:04 STEP 1278 H= 319:30 +CPU= 30.916
  00:00:35 STEP 1279 H= 319:45 +CPU= 30.376
  00:01:07 STEP 1280 H= 320:00 +CPU= 31.934
  00:01:38 STEP 1281 H= 320:15 +CPU= 31.060
  00:02:10 STEP 1282 H= 320:30 +CPU= 31.268
  00:02:41 STEP 1283 H= 320:45 +CPU= 31.562
  00:03:21 STEP 1284 H= 321:00 +CPU= 38.933
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001284
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001284
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001284
  00:03:52 STEP 1285 H= 321:15 +CPU= 30.771
  00:04:22 STEP 1286 H= 321:30 +CPU= 30.444
  00:04:54 STEP 1287 H= 321:45 +CPU= 31.534
  00:05:26 STEP 1288 H= 322:00 +CPU= 32.079
  00:05:58 STEP 1289 H= 322:15 +CPU= 31.908
  00:06:31 STEP 1290 H= 322:30 +CPU= 31.963
  00:07:02 STEP 1291 H= 322:45 +CPU= 31.375
  00:07:32 STEP 1292 H= 323:00 +CPU= 29.961
  00:08:03 STEP 1293 H= 323:15 +CPU= 30.465
  00:08:32 STEP 1294 H= 323:30 +CPU= 29.417
  00:09:03 STEP 1295 H= 323:45 +CPU= 30.513
  00:09:46 STEP 1296 H= 324:00 +CPU= 42.723
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001296
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001296
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001296
  00:10:15 STEP 1297 H= 324:15 +CPU= 29.742
  00:10:46 STEP 1298 H= 324:30 +CPU= 30.169
  00:11:15 STEP 1299 H= 324:45 +CPU= 28.786
  00:11:45 STEP 1300 H= 325:00 +CPU= 30.026
  00:12:14 STEP 1301 H= 325:15 +CPU= 29.450
  00:12:44 STEP 1302 H= 325:30 +CPU= 29.924
  00:13:14 STEP 1303 H= 325:45 +CPU= 29.916
  00:13:45 STEP 1304 H= 326:00 +CPU= 31.040
  00:14:16 STEP 1305 H= 326:15 +CPU= 30.363
  00:14:48 STEP 1306 H= 326:30 +CPU= 31.605
  00:15:17 STEP 1307 H= 326:45 +CPU= 29.817
  00:15:55 STEP 1308 H= 327:00 +CPU= 37.642
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001308
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001308
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001308
  00:16:26 STEP 1309 H= 327:15 +CPU= 30.482
  00:16:56 STEP 1310 H= 327:30 +CPU= 29.740
  00:17:26 STEP 1311 H= 327:45 +CPU= 30.007
  00:17:56 STEP 1312 H= 328:00 +CPU= 30.373
  00:18:26 STEP 1313 H= 328:15 +CPU= 29.640
  00:18:56 STEP 1314 H= 328:30 +CPU= 30.356
  00:19:27 STEP 1315 H= 328:45 +CPU= 30.315
  00:19:58 STEP 1316 H= 329:00 +CPU= 31.357
  00:20:29 STEP 1317 H= 329:15 +CPU= 30.746
  00:20:59 STEP 1318 H= 329:30 +CPU= 29.733
  00:21:29 STEP 1319 H= 329:45 +CPU= 30.164
  00:22:07 STEP 1320 H= 330:00 +CPU= 37.300
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001320
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001320
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001320
  00:22:36 STEP 1321 H= 330:15 +CPU= 29.062
  00:23:07 STEP 1322 H= 330:30 +CPU= 31.299
  00:23:37 STEP 1323 H= 330:45 +CPU= 29.754
  00:24:08 STEP 1324 H= 331:00 +CPU= 30.485
  00:24:38 STEP 1325 H= 331:15 +CPU= 30.404
  00:25:08 STEP 1326 H= 331:30 +CPU= 29.577
  00:25:38 STEP 1327 H= 331:45 +CPU= 30.089
  00:26:09 STEP 1328 H= 332:00 +CPU= 30.570
  00:26:40 STEP 1329 H= 332:15 +CPU= 30.856
  00:27:09 STEP 1330 H= 332:30 +CPU= 29.245
  00:27:40 STEP 1331 H= 332:45 +CPU= 30.552
  00:28:17 STEP 1332 H= 333:00 +CPU= 37.567
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001332
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001332
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001332
  00:28:48 STEP 1333 H= 333:15 +CPU= 30.623
  00:29:18 STEP 1334 H= 333:30 +CPU= 30.301
  00:29:48 STEP 1335 H= 333:45 +CPU= 29.747
  00:30:20 STEP 1336 H= 334:00 +CPU= 31.716
  00:30:49 STEP 1337 H= 334:15 +CPU= 28.932
  00:31:19 STEP 1338 H= 334:30 +CPU= 30.015
  00:31:49 STEP 1339 H= 334:45 +CPU= 29.598
  00:32:20 STEP 1340 H= 335:00 +CPU= 30.905
  00:32:50 STEP 1341 H= 335:15 +CPU= 30.219
  00:33:19 STEP 1342 H= 335:30 +CPU= 28.783
  00:33:49 STEP 1343 H= 335:45 +CPU= 30.091
  00:34:31 STEP 1344 H= 336:00 +CPU= 41.550
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001344
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001344
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001344
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001248
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001248
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001248
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001260
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001260
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001260
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001272
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001272
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001272
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001284
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001284
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001284
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001296
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001296
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001296
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001308
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001308
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001308
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001320
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001320
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001320
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001332
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001332
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001332
Zipping up the intermediate file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_13.zip
Uploading the intermediate file: upload_file_13.zip
  00:35:01 STEP 1345 H= 336:15 +CPU= 30.169
  00:35:32 STEP 1346 H= 336:30 +CPU= 30.259
  00:36:02 STEP 1347 H= 336:45 +CPU= 30.414
  00:36:33 STEP 1348 H= 337:00 +CPU= 30.574
  00:37:03 STEP 1349 H= 337:15 +CPU= 29.635
  00:37:34 STEP 1350 H= 337:30 +CPU= 30.773
  00:38:04 STEP 1351 H= 337:45 +CPU= 29.987
  00:38:34 STEP 1352 H= 338:00 +CPU= 30.500
  00:39:05 STEP 1353 H= 338:15 +CPU= 30.324
  00:39:34 STEP 1354 H= 338:30 +CPU= 29.179
  00:40:03 STEP 1355 H= 338:45 +CPU= 29.015
  00:40:40 STEP 1356 H= 339:00 +CPU= 36.654
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001356
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001356
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001356
  00:41:10 STEP 1357 H= 339:15 +CPU= 29.661
  00:41:40 STEP 1358 H= 339:30 +CPU= 29.830
  00:42:09 STEP 1359 H= 339:45 +CPU= 29.563
  00:42:40 STEP 1360 H= 340:00 +CPU= 30.660
  00:43:10 STEP 1361 H= 340:15 +CPU= 30.480
  00:43:41 STEP 1362 H= 340:30 +CPU= 30.430
  00:44:11 STEP 1363 H= 340:45 +CPU= 29.894
  00:44:41 STEP 1364 H= 341:00 +CPU= 30.272
  00:45:12 STEP 1365 H= 341:15 +CPU= 30.706
  00:45:42 STEP 1366 H= 341:30 +CPU= 29.393
  00:46:12 STEP 1367 H= 341:45 +CPU= 30.376
  00:46:50 STEP 1368 H= 342:00 +CPU= 37.966
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001368
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001368
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001368
  00:47:20 STEP 1369 H= 342:15 +CPU= 29.674
  00:47:50 STEP 1370 H= 342:30 +CPU= 30.457
  00:48:21 STEP 1371 H= 342:45 +CPU= 30.264
  00:48:51 STEP 1372 H= 343:00 +CPU= 30.380
  00:49:21 STEP 1373 H= 343:15 +CPU= 29.812
  00:49:52 STEP 1374 H= 343:30 +CPU= 30.211
  00:50:22 STEP 1375 H= 343:45 +CPU= 30.032
  00:50:53 STEP 1376 H= 344:00 +CPU= 30.881
  00:51:23 STEP 1377 H= 344:15 +CPU= 30.341
  00:51:53 STEP 1378 H= 344:30 +CPU= 29.957
  00:52:24 STEP 1379 H= 344:45 +CPU= 30.218
  00:53:02 STEP 1380 H= 345:00 +CPU= 38.550
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001380
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001380
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001380
  00:53:32 STEP 1381 H= 345:15 +CPU= 29.848
  00:54:03 STEP 1382 H= 345:30 +CPU= 30.537
  00:54:33 STEP 1383 H= 345:45 +CPU= 30.138
  00:55:04 STEP 1384 H= 346:00 +CPU= 31.156
  00:55:35 STEP 1385 H= 346:15 +CPU= 30.182
  00:56:05 STEP 1386 H= 346:30 +CPU= 30.258
  00:56:34 STEP 1387 H= 346:45 +CPU= 29.403
  00:57:04 STEP 1388 H= 347:00 +CPU= 29.720
  00:57:34 STEP 1389 H= 347:15 +CPU= 30.006
  00:58:04 STEP 1390 H= 347:30 +CPU= 29.429
  00:58:34 STEP 1391 H= 347:45 +CPU= 30.552
  00:59:17 STEP 1392 H= 348:00 +CPU= 41.905
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001392
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001392
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001392
  00:59:47 STEP 1393 H= 348:15 +CPU= 30.033
  01:00:16 STEP 1394 H= 348:30 +CPU= 29.443
  01:00:45 STEP 1395 H= 348:45 +CPU= 28.894
  01:01:16 STEP 1396 H= 349:00 +CPU= 30.520
  01:01:45 STEP 1397 H= 349:15 +CPU= 29.464
  01:02:15 STEP 1398 H= 349:30 +CPU= 28.968
  01:02:45 STEP 1399 H= 349:45 +CPU= 29.949
  01:03:15 STEP 1400 H= 350:00 +CPU= 30.564
  01:03:46 STEP 1401 H= 350:15 +CPU= 30.149
  01:04:16 STEP 1402 H= 350:30 +CPU= 30.320
  01:04:47 STEP 1403 H= 350:45 +CPU= 31.197
  01:05:25 STEP 1404 H= 351:00 +CPU= 37.764
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001404
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001404
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001404
  01:05:55 STEP 1405 H= 351:15 +CPU= 29.705
  01:06:25 STEP 1406 H= 351:30 +CPU= 29.957
  01:06:55 STEP 1407 H= 351:45 +CPU= 29.762
  01:07:27 STEP 1408 H= 352:00 +CPU= 31.630
  01:07:57 STEP 1409 H= 352:15 +CPU= 29.808
  01:08:28 STEP 1410 H= 352:30 +CPU= 30.902
  01:08:57 STEP 1411 H= 352:45 +CPU= 29.769
  01:09:28 STEP 1412 H= 353:00 +CPU= 30.776
  01:10:00 STEP 1413 H= 353:15 +CPU= 31.561
  01:10:29 STEP 1414 H= 353:30 +CPU= 29.328
  01:11:00 STEP 1415 H= 353:45 +CPU= 30.880
  01:11:38 STEP 1416 H= 354:00 +CPU= 37.438
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001416
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001416
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001416
  01:12:09 STEP 1417 H= 354:15 +CPU= 30.639
  01:12:39 STEP 1418 H= 354:30 +CPU= 29.962
  01:13:09 STEP 1419 H= 354:45 +CPU= 29.989
  01:13:40 STEP 1420 H= 355:00 +CPU= 30.847
  01:14:09 STEP 1421 H= 355:15 +CPU= 29.020
  01:14:39 STEP 1422 H= 355:30 +CPU= 29.558
  01:15:09 STEP 1423 H= 355:45 +CPU= 30.610
  01:15:41 STEP 1424 H= 356:00 +CPU= 31.239
  01:16:11 STEP 1425 H= 356:15 +CPU= 30.080
  01:16:40 STEP 1426 H= 356:30 +CPU= 29.567
  01:17:10 STEP 1427 H= 356:45 +CPU= 29.644
  01:17:48 STEP 1428 H= 357:00 +CPU= 38.002
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001428
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001428
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001428
  01:18:18 STEP 1429 H= 357:15 +CPU= 29.962
  01:18:48 STEP 1430 H= 357:30 +CPU= 29.706
  01:19:18 STEP 1431 H= 357:45 +CPU= 30.024
  01:19:49 STEP 1432 H= 358:00 +CPU= 30.474
  01:20:19 STEP 1433 H= 358:15 +CPU= 30.531
  01:20:50 STEP 1434 H= 358:30 +CPU= 30.745
  01:21:20 STEP 1435 H= 358:45 +CPU= 29.622
  01:21:52 STEP 1436 H= 359:00 +CPU= 31.952
  01:22:23 STEP 1437 H= 359:15 +CPU= 30.605
  01:22:52 STEP 1438 H= 359:30 +CPU= 29.538
  01:23:23 STEP 1439 H= 359:45 +CPU= 30.283
  01:24:02 STEP 1440 H= 360:00 +CPU= 38.949
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.8387536113E-21      I       0.3366727796E-05      I
ICloud fractionI      0.0000000000E+00      I       0.3680498991E-01      I
I  Relat. Hum. I      0.3787830210E+00      I       0.4188727091E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3652890064E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    328548721 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS547581.2 Mbytes/s, TOTAL TIME=    0.0106( 5.7%)
 01:24:02 STEP 1440 H= 360:00 +CPU= 38.949
 END CNT3
 NSTEP =  1440 CNT0   000000000  27.185  27.185  29.225   0.930               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ            0/           0 WRITTEN        42886/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    758    0        0            0    42886 ************    97.3     0.0    97.3     0.0   264.2
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 *********       0.0    100.00   -100.00
   1 CNT4     - FORWARD INTEGRATION                  1 *********       0.0    100.01   -100.01
   8 SCAN2M - GRID-POINT DYNAMICS                 1562    5588.3    5588.3      0.00      0.00
   9 SPCM     - SPECTRAL COMP.                    1440    1084.8    1084.8      0.00      0.00
  10 SCAN2M - PHYSICS                             1441   15690.3   15690.3      0.00      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                121    1148.0    1148.0      0.00      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 63      54.9      54.9      0.00      0.00
  14 SUINIF                                          1    3664.2    3664.2      0.00      0.00
  17 GRIDFPOS IN CNT4                              121      33.2      33.2      0.00      0.00
  18 SUSPECG                                         1    2003.4    2003.4      0.00      0.00
  19 SUSPEC                                          1    2009.8    2009.8      0.00      0.00
  24 SUGRIDU                                         1    1223.8    1223.8      0.00      0.00
  25 SPECRT                                          1     308.4     308.4      0.00      0.00
  26 SUGRIDF                                         1     122.2     122.2      0.00      0.00
  27 RESTART FILES - WRITING                        30    4628.9    4628.9      0.00      0.00
  28 RESTART FILES - READING                         1       0.0       0.0      0.00      0.00
  29 SU4FPOS IN CNT4                               121       0.6       0.6      0.00      0.00
  30 DYNFPOS IN CNT4                               121 *********       0.0    139.41   -139.41
  31 POSDDH IN STEPO                                 7     300.7     300.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                   1441    5771.8    5771.8      0.00      0.00
  39 SU0YOMB                                         1     474.0     474.0      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                   1441      21.2      21.2      0.00      0.00
  54 SPCM     - M TO S/S TO M TRANSP.             1440     351.8     351.8      0.00      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.             1440      80.9      80.9      0.00      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 63       0.5       0.5      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM     3125     479.0     479.0      0.00      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM      3006     278.4     278.4      0.00      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM       3006     106.0     106.0      0.00      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM      3125     152.5     152.5      0.00      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      44.2      44.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION           3125      70.0      70.0      0.00      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION           3006      37.5      37.5      0.00      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION           3125      90.7      90.7      0.00      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION           3006      31.9      31.9      0.00      0.00
 400 GSTATS                                     167374       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                155706       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME  -116009.5 CPU TIME   45577.6 VECTOR TIME    45577.6


===-=== END   OF TIMING STATISTICS ===-===
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /home/tommcfarland/slots/1/ICMGGh7zg+001440
Moving to projects directory: /home/tommcfarland/slots/1/ICMSHh7zg+001440
Moving to projects directory: /home/tommcfarland/slots/1/ICMUAh7zg+001440
Adding to the zip: /home/tommcfarland/slots/1/NODE.001_01
Adding to the zip: /home/tommcfarland/slots/1/ifs.stat
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001368
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001404
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001428
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001404
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001380
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001356
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001356
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001440
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001344
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001428
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001428
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001368
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001440
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001416
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001392
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001344
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001356
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001368
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001344
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001404
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001416
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001416
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001380
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001392
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMUAh7zg+001380
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMSHh7zg+001392
Adding to the zip: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_12209860/ICMGGh7zg+001440
Zipping up the final file: /home/tommcfarland/projects/climateprediction.net/oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1_r1101822317_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
01:27:36 (231687): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:56:00 1526341 22314731 oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1 1,295,100 45,929 0.0355
09 May 2023 16:56:00 1526341 22314731 oifs_43r3_bl_a2dd_2016092300_15_991_12209860_1 864,000 30,818 0.0357


©2024 cpdn.org