climateprediction.net home page
Task 22434366

Task 22434366

Name oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0
Workunit 12282797
Created 7 Jun 2024, 12:14:59 UTC
Sent 7 Jun 2024, 15:06:40 UTC
Report deadline 6 Aug 2024, 15:06:40 UTC
Received 8 Jun 2024, 19:22:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1472385
Run time 11 hours 31 min 21 sec
CPU time 11 hours 31 min 21 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 3.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 4,994.71 MB
Peak swap size 5,980.78 MB
Peak disk usage 1,378.41 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
990 H= 247:30 +CPU= 26.330
  15:56:54 STEP  991 H= 247:45 +CPU= 27.896
  15:57:22 STEP  992 H= 248:00 +CPU= 28.303
  15:57:48 STEP  993 H= 248:15 +CPU= 25.914
  15:58:17 STEP  994 H= 248:30 +CPU= 28.968
  15:58:41 STEP  995 H= 248:45 +CPU= 24.207
  15:59:12 STEP  996 H= 249:00 +CPU= 30.864
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+000996
  15:59:39 STEP  997 H= 249:15 +CPU= 27.225
  16:00:07 STEP  998 H= 249:30 +CPU= 27.294
  16:00:34 STEP  999 H= 249:45 +CPU= 26.695
  16:01:01 STEP 1000 H= 250:00 +CPU= 27.575
  16:01:30 STEP 1001 H= 250:15 +CPU= 28.431
  16:01:56 STEP 1002 H= 250:30 +CPU= 26.411
  16:02:25 STEP 1003 H= 250:45 +CPU= 28.643
  16:02:54 STEP 1004 H= 251:00 +CPU= 28.732
  16:03:21 STEP 1005 H= 251:15 +CPU= 27.376
  16:03:46 STEP 1006 H= 251:30 +CPU= 25.111
  16:04:15 STEP 1007 H= 251:45 +CPU= 28.664
  16:04:52 STEP 1008 H= 252:00 +CPU= 36.837
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001008
  16:05:19 STEP 1009 H= 252:15 +CPU= 26.310
  16:05:46 STEP 1010 H= 252:30 +CPU= 27.336
  16:06:12 STEP 1011 H= 252:45 +CPU= 26.335
  16:06:40 STEP 1012 H= 253:00 +CPU= 27.634
  16:07:06 STEP 1013 H= 253:15 +CPU= 25.907
  16:07:35 STEP 1014 H= 253:30 +CPU= 28.704
  16:08:05 STEP 1015 H= 253:45 +CPU= 30.052
  16:08:33 STEP 1016 H= 254:00 +CPU= 28.146
  16:09:01 STEP 1017 H= 254:15 +CPU= 27.636
  16:09:30 STEP 1018 H= 254:30 +CPU= 29.008
  16:09:57 STEP 1019 H= 254:45 +CPU= 26.819
  16:10:33 STEP 1020 H= 255:00 +CPU= 36.275
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001020
  16:11:00 STEP 1021 H= 255:15 +CPU= 26.197
  16:11:29 STEP 1022 H= 255:30 +CPU= 28.799
  16:11:54 STEP 1023 H= 255:45 +CPU= 25.192
  16:12:18 STEP 1024 H= 256:00 +CPU= 24.350
  16:12:43 STEP 1025 H= 256:15 +CPU= 24.930
  16:13:07 STEP 1026 H= 256:30 +CPU= 23.590
  16:13:32 STEP 1027 H= 256:45 +CPU= 25.471
  16:14:00 STEP 1028 H= 257:00 +CPU= 27.536
  16:14:25 STEP 1029 H= 257:15 +CPU= 25.279
  16:14:56 STEP 1030 H= 257:30 +CPU= 30.261
  16:15:20 STEP 1031 H= 257:45 +CPU= 24.345
  16:15:51 STEP 1032 H= 258:00 +CPU= 30.725
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001032
  16:16:20 STEP 1033 H= 258:15 +CPU= 28.940
  16:16:44 STEP 1034 H= 258:30 +CPU= 24.131
  16:17:11 STEP 1035 H= 258:45 +CPU= 27.241
  16:17:37 STEP 1036 H= 259:00 +CPU= 25.396
  16:18:02 STEP 1037 H= 259:15 +CPU= 25.156
  16:18:30 STEP 1038 H= 259:30 +CPU= 27.818
  16:18:55 STEP 1039 H= 259:45 +CPU= 25.814
  16:19:23 STEP 1040 H= 260:00 +CPU= 27.515
  16:19:47 STEP 1041 H= 260:15 +CPU= 24.283
  16:20:14 STEP 1042 H= 260:30 +CPU= 26.916
  16:20:40 STEP 1043 H= 260:45 +CPU= 25.570
  16:21:12 STEP 1044 H= 261:00 +CPU= 31.449
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001044
  16:21:38 STEP 1045 H= 261:15 +CPU= 26.853
  16:22:05 STEP 1046 H= 261:30 +CPU= 26.854
  16:22:33 STEP 1047 H= 261:45 +CPU= 27.804
  16:23:05 STEP 1048 H= 262:00 +CPU= 32.080
  16:23:31 STEP 1049 H= 262:15 +CPU= 25.466
  16:23:59 STEP 1050 H= 262:30 +CPU= 28.304
  16:24:26 STEP 1051 H= 262:45 +CPU= 27.329
  16:24:53 STEP 1052 H= 263:00 +CPU= 27.114
  16:25:20 STEP 1053 H= 263:15 +CPU= 26.162
  16:25:46 STEP 1054 H= 263:30 +CPU= 26.020
  16:26:12 STEP 1055 H= 263:45 +CPU= 26.486
  16:26:49 STEP 1056 H= 264:00 +CPU= 36.542
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_10.zip
Uploading the intermediate file: upload_file_10.zip
  16:27:14 STEP 1057 H= 264:15 +CPU= 25.423
  16:27:39 STEP 1058 H= 264:30 +CPU= 24.592
  16:28:09 STEP 1059 H= 264:45 +CPU= 29.856
  16:28:36 STEP 1060 H= 265:00 +CPU= 26.735
  16:29:04 STEP 1061 H= 265:15 +CPU= 28.086
  16:29:28 STEP 1062 H= 265:30 +CPU= 24.568
  16:29:58 STEP 1063 H= 265:45 +CPU= 29.646
  16:30:25 STEP 1064 H= 266:00 +CPU= 26.677
  16:30:51 STEP 1065 H= 266:15 +CPU= 25.743
  16:31:19 STEP 1066 H= 266:30 +CPU= 28.569
  16:31:51 STEP 1067 H= 266:45 +CPU= 31.655
  16:32:27 STEP 1068 H= 267:00 +CPU= 35.481
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001068
  16:32:53 STEP 1069 H= 267:15 +CPU= 26.296
  16:33:20 STEP 1070 H= 267:30 +CPU= 27.131
  16:33:46 STEP 1071 H= 267:45 +CPU= 25.748
  16:34:15 STEP 1072 H= 268:00 +CPU= 29.176
  16:34:43 STEP 1073 H= 268:15 +CPU= 27.735
  16:35:10 STEP 1074 H= 268:30 +CPU= 26.987
  16:35:39 STEP 1075 H= 268:45 +CPU= 28.760
  16:36:05 STEP 1076 H= 269:00 +CPU= 25.958
  16:36:34 STEP 1077 H= 269:15 +CPU= 29.486
  16:37:03 STEP 1078 H= 269:30 +CPU= 28.213
  16:37:31 STEP 1079 H= 269:45 +CPU= 28.429
  16:38:07 STEP 1080 H= 270:00 +CPU= 35.416
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001080
  16:38:34 STEP 1081 H= 270:15 +CPU= 26.804
  16:38:58 STEP 1082 H= 270:30 +CPU= 24.701
  16:39:27 STEP 1083 H= 270:45 +CPU= 28.861
  16:39:55 STEP 1084 H= 271:00 +CPU= 27.805
  16:40:23 STEP 1085 H= 271:15 +CPU= 27.474
  16:40:47 STEP 1086 H= 271:30 +CPU= 24.724
  16:41:13 STEP 1087 H= 271:45 +CPU= 25.501
  16:41:37 STEP 1088 H= 272:00 +CPU= 24.490
  16:42:04 STEP 1089 H= 272:15 +CPU= 26.135
  16:42:32 STEP 1090 H= 272:30 +CPU= 28.756
  16:42:57 STEP 1091 H= 272:45 +CPU= 24.898
  16:43:29 STEP 1092 H= 273:00 +CPU= 31.481
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001092
  16:43:54 STEP 1093 H= 273:15 +CPU= 25.238
  16:44:22 STEP 1094 H= 273:30 +CPU= 27.697
  16:44:45 STEP 1095 H= 273:45 +CPU= 23.635
  16:45:13 STEP 1096 H= 274:00 +CPU= 27.617
  16:45:42 STEP 1097 H= 274:15 +CPU= 28.247
  16:46:07 STEP 1098 H= 274:30 +CPU= 25.104
  16:46:32 STEP 1099 H= 274:45 +CPU= 25.307
  16:47:04 STEP 1100 H= 275:00 +CPU= 31.862
  16:47:28 STEP 1101 H= 275:15 +CPU= 24.597
  16:47:57 STEP 1102 H= 275:30 +CPU= 28.607
  16:48:25 STEP 1103 H= 275:45 +CPU= 27.733
  16:49:03 STEP 1104 H= 276:00 +CPU= 38.416
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001104
  16:49:29 STEP 1105 H= 276:15 +CPU= 25.937
  16:49:54 STEP 1106 H= 276:30 +CPU= 24.775
  16:50:20 STEP 1107 H= 276:45 +CPU= 25.269
  16:50:48 STEP 1108 H= 277:00 +CPU= 28.383
  16:51:14 STEP 1109 H= 277:15 +CPU= 25.805
  16:51:42 STEP 1110 H= 277:30 +CPU= 28.151
  16:52:09 STEP 1111 H= 277:45 +CPU= 27.462
  16:52:35 STEP 1112 H= 278:00 +CPU= 25.912
  16:53:01 STEP 1113 H= 278:15 +CPU= 25.975
  16:53:28 STEP 1114 H= 278:30 +CPU= 27.069
  16:53:53 STEP 1115 H= 278:45 +CPU= 24.803
  16:54:28 STEP 1116 H= 279:00 +CPU= 34.784
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001116
  16:54:55 STEP 1117 H= 279:15 +CPU= 27.318
  16:55:22 STEP 1118 H= 279:30 +CPU= 26.332
  16:55:46 STEP 1119 H= 279:45 +CPU= 23.523
  16:56:11 STEP 1120 H= 280:00 +CPU= 25.666
  16:56:36 STEP 1121 H= 280:15 +CPU= 24.892
  16:57:02 STEP 1122 H= 280:30 +CPU= 25.359
  16:57:31 STEP 1123 H= 280:45 +CPU= 29.835
  16:58:01 STEP 1124 H= 281:00 +CPU= 29.288
  16:58:28 STEP 1125 H= 281:15 +CPU= 27.193
  16:58:53 STEP 1126 H= 281:30 +CPU= 25.372
  16:59:22 STEP 1127 H= 281:45 +CPU= 28.223
  16:59:53 STEP 1128 H= 282:00 +CPU= 31.671
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001128
  17:00:19 STEP 1129 H= 282:15 +CPU= 25.676
  17:00:48 STEP 1130 H= 282:30 +CPU= 29.116
  17:01:16 STEP 1131 H= 282:45 +CPU= 27.478
  17:01:43 STEP 1132 H= 283:00 +CPU= 27.000
  17:02:08 STEP 1133 H= 283:15 +CPU= 24.654
  17:02:35 STEP 1134 H= 283:30 +CPU= 26.932
  17:02:58 STEP 1135 H= 283:45 +CPU= 23.613
  17:03:27 STEP 1136 H= 284:00 +CPU= 28.223
  17:03:53 STEP 1137 H= 284:15 +CPU= 26.104
  17:04:20 STEP 1138 H= 284:30 +CPU= 27.361
  17:04:47 STEP 1139 H= 284:45 +CPU= 26.550
  17:05:21 STEP 1140 H= 285:00 +CPU= 33.935
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001140
  17:05:48 STEP 1141 H= 285:15 +CPU= 27.411
  17:06:15 STEP 1142 H= 285:30 +CPU= 26.345
  17:06:44 STEP 1143 H= 285:45 +CPU= 28.622
  17:07:13 STEP 1144 H= 286:00 +CPU= 28.898
  17:07:41 STEP 1145 H= 286:15 +CPU= 28.583
  17:08:10 STEP 1146 H= 286:30 +CPU= 28.887
  17:08:39 STEP 1147 H= 286:45 +CPU= 28.680
  17:09:09 STEP 1148 H= 287:00 +CPU= 29.946
  17:09:34 STEP 1149 H= 287:15 +CPU= 24.641
  17:09:58 STEP 1150 H= 287:30 +CPU= 24.328
  17:10:26 STEP 1151 H= 287:45 +CPU= 27.638
  17:11:01 STEP 1152 H= 288:00 +CPU= 34.523
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_11.zip
Uploading the intermediate file: upload_file_11.zip
  17:11:29 STEP 1153 H= 288:15 +CPU= 28.563
  17:11:56 STEP 1154 H= 288:30 +CPU= 27.003
  17:12:25 STEP 1155 H= 288:45 +CPU= 28.836
  17:12:52 STEP 1156 H= 289:00 +CPU= 27.153
  17:13:19 STEP 1157 H= 289:15 +CPU= 26.722
  17:13:47 STEP 1158 H= 289:30 +CPU= 27.980
  17:14:15 STEP 1159 H= 289:45 +CPU= 27.764
  17:14:42 STEP 1160 H= 290:00 +CPU= 27.443
  17:15:10 STEP 1161 H= 290:15 +CPU= 27.757
  17:15:39 STEP 1162 H= 290:30 +CPU= 28.137
  17:16:05 STEP 1163 H= 290:45 +CPU= 26.462
  17:16:35 STEP 1164 H= 291:00 +CPU= 29.796
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001164
  17:17:00 STEP 1165 H= 291:15 +CPU= 24.932
  17:17:26 STEP 1166 H= 291:30 +CPU= 26.491
  17:17:57 STEP 1167 H= 291:45 +CPU= 30.361
  17:18:27 STEP 1168 H= 292:00 +CPU= 30.176
  17:18:55 STEP 1169 H= 292:15 +CPU= 28.372
  17:19:22 STEP 1170 H= 292:30 +CPU= 26.694
  17:19:48 STEP 1171 H= 292:45 +CPU= 26.070
  17:20:15 STEP 1172 H= 293:00 +CPU= 26.052
  17:20:40 STEP 1173 H= 293:15 +CPU= 25.852
  17:21:06 STEP 1174 H= 293:30 +CPU= 25.955
  17:21:31 STEP 1175 H= 293:45 +CPU= 24.938
  17:22:05 STEP 1176 H= 294:00 +CPU= 33.795
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001176
  17:22:35 STEP 1177 H= 294:15 +CPU= 29.496
  17:23:03 STEP 1178 H= 294:30 +CPU= 28.757
  17:23:33 STEP 1179 H= 294:45 +CPU= 29.448
  17:24:01 STEP 1180 H= 295:00 +CPU= 28.141
  17:24:27 STEP 1181 H= 295:15 +CPU= 25.451
  17:24:54 STEP 1182 H= 295:30 +CPU= 27.044
  17:25:21 STEP 1183 H= 295:45 +CPU= 27.202
  17:25:48 STEP 1184 H= 296:00 +CPU= 26.968
  17:26:14 STEP 1185 H= 296:15 +CPU= 26.139
  17:26:40 STEP 1186 H= 296:30 +CPU= 25.597
  17:27:06 STEP 1187 H= 296:45 +CPU= 26.438
  17:27:36 STEP 1188 H= 297:00 +CPU= 29.394
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001188
  17:28:01 STEP 1189 H= 297:15 +CPU= 25.304
  17:28:29 STEP 1190 H= 297:30 +CPU= 27.780
  17:28:54 STEP 1191 H= 297:45 +CPU= 25.588
  17:29:19 STEP 1192 H= 298:00 +CPU= 24.172
  17:29:44 STEP 1193 H= 298:15 +CPU= 25.418
  17:30:12 STEP 1194 H= 298:30 +CPU= 27.627
  17:30:37 STEP 1195 H= 298:45 +CPU= 25.220
  17:31:04 STEP 1196 H= 299:00 +CPU= 27.155
  17:31:33 STEP 1197 H= 299:15 +CPU= 28.272
  17:32:02 STEP 1198 H= 299:30 +CPU= 29.344
  17:32:30 STEP 1199 H= 299:45 +CPU= 28.294
  17:33:06 STEP 1200 H= 300:00 +CPU= 35.595
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001200
  17:33:33 STEP 1201 H= 300:15 +CPU= 27.018
  17:34:00 STEP 1202 H= 300:30 +CPU= 26.483
  17:34:28 STEP 1203 H= 300:45 +CPU= 27.906
  17:34:57 STEP 1204 H= 301:00 +CPU= 29.111
  17:35:22 STEP 1205 H= 301:15 +CPU= 25.218
  17:35:49 STEP 1206 H= 301:30 +CPU= 26.785
  17:36:16 STEP 1207 H= 301:45 +CPU= 26.235
  17:36:46 STEP 1208 H= 302:00 +CPU= 29.930
  17:37:15 STEP 1209 H= 302:15 +CPU= 29.464
  17:37:41 STEP 1210 H= 302:30 +CPU= 25.901
  17:38:10 STEP 1211 H= 302:45 +CPU= 28.402
  17:38:44 STEP 1212 H= 303:00 +CPU= 33.689
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001212
  17:39:08 STEP 1213 H= 303:15 +CPU= 24.545
  17:39:33 STEP 1214 H= 303:30 +CPU= 25.059
  17:40:00 STEP 1215 H= 303:45 +CPU= 26.688
  17:40:25 STEP 1216 H= 304:00 +CPU= 25.161
  17:40:52 STEP 1217 H= 304:15 +CPU= 26.861
  17:41:17 STEP 1218 H= 304:30 +CPU= 24.458
  17:41:46 STEP 1219 H= 304:45 +CPU= 28.965
  17:42:12 STEP 1220 H= 305:00 +CPU= 26.012
  17:42:40 STEP 1221 H= 305:15 +CPU= 28.633
  17:43:10 STEP 1222 H= 305:30 +CPU= 29.103
  17:43:35 STEP 1223 H= 305:45 +CPU= 25.621
  17:44:11 STEP 1224 H= 306:00 +CPU= 35.458
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001224
  17:44:38 STEP 1225 H= 306:15 +CPU= 27.148
  17:45:07 STEP 1226 H= 306:30 +CPU= 29.017
  17:45:33 STEP 1227 H= 306:45 +CPU= 25.924
  17:45:59 STEP 1228 H= 307:00 +CPU= 26.386
  17:46:28 STEP 1229 H= 307:15 +CPU= 27.926
  17:46:51 STEP 1230 H= 307:30 +CPU= 23.578
  17:47:17 STEP 1231 H= 307:45 +CPU= 26.244
  17:47:46 STEP 1232 H= 308:00 +CPU= 28.952
  17:48:12 STEP 1233 H= 308:15 +CPU= 26.020
  17:48:39 STEP 1234 H= 308:30 +CPU= 26.587
  17:49:05 STEP 1235 H= 308:45 +CPU= 25.831
  17:49:38 STEP 1236 H= 309:00 +CPU= 32.858
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001236
  17:50:04 STEP 1237 H= 309:15 +CPU= 26.304
  17:50:31 STEP 1238 H= 309:30 +CPU= 26.181
  17:50:58 STEP 1239 H= 309:45 +CPU= 27.514
  17:51:28 STEP 1240 H= 310:00 +CPU= 29.407
  17:51:55 STEP 1241 H= 310:15 +CPU= 26.979
  17:52:21 STEP 1242 H= 310:30 +CPU= 26.050
  17:52:47 STEP 1243 H= 310:45 +CPU= 26.398
  17:53:17 STEP 1244 H= 311:00 +CPU= 29.349
  17:53:47 STEP 1245 H= 311:15 +CPU= 30.073
  17:54:12 STEP 1246 H= 311:30 +CPU= 25.397
  17:54:36 STEP 1247 H= 311:45 +CPU= 23.906
  17:55:12 STEP 1248 H= 312:00 +CPU= 35.601
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_12.zip
Uploading the intermediate file: upload_file_12.zip
  17:55:40 STEP 1249 H= 312:15 +CPU= 28.142
Uploading trickle at timestep: 1123200
  17:56:08 STEP 1250 H= 312:30 +CPU= 27.868
  17:56:33 STEP 1251 H= 312:45 +CPU= 24.986
  17:57:00 STEP 1252 H= 313:00 +CPU= 27.497
  17:57:26 STEP 1253 H= 313:15 +CPU= 25.590
  17:57:53 STEP 1254 H= 313:30 +CPU= 26.403
  17:58:16 STEP 1255 H= 313:45 +CPU= 23.358
  17:58:43 STEP 1256 H= 314:00 +CPU= 26.611
  17:59:10 STEP 1257 H= 314:15 +CPU= 27.748
  17:59:36 STEP 1258 H= 314:30 +CPU= 25.349
  18:00:04 STEP 1259 H= 314:45 +CPU= 28.024
  18:00:37 STEP 1260 H= 315:00 +CPU= 32.513
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001260
  18:01:06 STEP 1261 H= 315:15 +CPU= 29.909
  18:01:34 STEP 1262 H= 315:30 +CPU= 27.364
  18:02:03 STEP 1263 H= 315:45 +CPU= 28.519
  18:02:33 STEP 1264 H= 316:00 +CPU= 29.318
  18:03:01 STEP 1265 H= 316:15 +CPU= 27.863
  18:03:28 STEP 1266 H= 316:30 +CPU= 27.671
  18:03:55 STEP 1267 H= 316:45 +CPU= 26.534
  18:04:26 STEP 1268 H= 317:00 +CPU= 30.272
  18:04:49 STEP 1269 H= 317:15 +CPU= 23.840
  18:05:14 STEP 1270 H= 317:30 +CPU= 24.209
  18:05:41 STEP 1271 H= 317:45 +CPU= 26.565
  18:06:11 STEP 1272 H= 318:00 +CPU= 30.021
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001272
  18:06:33 STEP 1273 H= 318:15 +CPU= 22.317
  18:06:58 STEP 1274 H= 318:30 +CPU= 24.981
  18:07:21 STEP 1275 H= 318:45 +CPU= 23.496
  18:07:46 STEP 1276 H= 319:00 +CPU= 24.155
  18:08:12 STEP 1277 H= 319:15 +CPU= 26.616
  18:08:37 STEP 1278 H= 319:30 +CPU= 25.329
  18:09:01 STEP 1279 H= 319:45 +CPU= 23.670
  18:09:26 STEP 1280 H= 320:00 +CPU= 24.746
  18:09:51 STEP 1281 H= 320:15 +CPU= 24.567
  18:10:18 STEP 1282 H= 320:30 +CPU= 27.675
  18:10:43 STEP 1283 H= 320:45 +CPU= 24.207
  18:11:12 STEP 1284 H= 321:00 +CPU= 29.442
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001284
  18:11:37 STEP 1285 H= 321:15 +CPU= 24.992
  18:12:05 STEP 1286 H= 321:30 +CPU= 27.762
  18:12:31 STEP 1287 H= 321:45 +CPU= 25.900
  18:12:57 STEP 1288 H= 322:00 +CPU= 26.104
  18:13:23 STEP 1289 H= 322:15 +CPU= 25.687
  18:13:47 STEP 1290 H= 322:30 +CPU= 24.361
  18:14:12 STEP 1291 H= 322:45 +CPU= 25.471
  18:14:37 STEP 1292 H= 323:00 +CPU= 24.827
  18:15:02 STEP 1293 H= 323:15 +CPU= 25.048
  18:15:30 STEP 1294 H= 323:30 +CPU= 27.199
  18:15:57 STEP 1295 H= 323:45 +CPU= 26.942
  18:16:30 STEP 1296 H= 324:00 +CPU= 33.247
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001296
  18:16:57 STEP 1297 H= 324:15 +CPU= 27.582
  18:17:24 STEP 1298 H= 324:30 +CPU= 26.178
  18:17:51 STEP 1299 H= 324:45 +CPU= 26.890
  18:18:20 STEP 1300 H= 325:00 +CPU= 28.900
  18:18:45 STEP 1301 H= 325:15 +CPU= 25.858
  18:19:14 STEP 1302 H= 325:30 +CPU= 28.570
  18:19:41 STEP 1303 H= 325:45 +CPU= 26.613
  18:20:05 STEP 1304 H= 326:00 +CPU= 23.873
  18:20:30 STEP 1305 H= 326:15 +CPU= 25.538
  18:20:58 STEP 1306 H= 326:30 +CPU= 27.170
  18:21:22 STEP 1307 H= 326:45 +CPU= 24.380
  18:21:54 STEP 1308 H= 327:00 +CPU= 31.700
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001308
  18:22:20 STEP 1309 H= 327:15 +CPU= 26.257
  18:22:42 STEP 1310 H= 327:30 +CPU= 22.277
  18:23:09 STEP 1311 H= 327:45 +CPU= 26.950
  18:23:36 STEP 1312 H= 328:00 +CPU= 26.566
  18:24:03 STEP 1313 H= 328:15 +CPU= 26.995
  18:24:28 STEP 1314 H= 328:30 +CPU= 25.029
  18:24:51 STEP 1315 H= 328:45 +CPU= 23.264
  18:25:15 STEP 1316 H= 329:00 +CPU= 23.654
  18:25:42 STEP 1317 H= 329:15 +CPU= 26.559
  18:26:06 STEP 1318 H= 329:30 +CPU= 24.194
  18:26:31 STEP 1319 H= 329:45 +CPU= 25.617
  18:27:00 STEP 1320 H= 330:00 +CPU= 28.523
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001320
  18:27:24 STEP 1321 H= 330:15 +CPU= 23.998
  18:27:48 STEP 1322 H= 330:30 +CPU= 24.364
  18:28:14 STEP 1323 H= 330:45 +CPU= 25.529
  18:28:41 STEP 1324 H= 331:00 +CPU= 27.554
  18:29:06 STEP 1325 H= 331:15 +CPU= 24.972
  18:29:30 STEP 1326 H= 331:30 +CPU= 23.118
  18:29:52 STEP 1327 H= 331:45 +CPU= 22.767
  18:30:16 STEP 1328 H= 332:00 +CPU= 23.882
  18:30:40 STEP 1329 H= 332:15 +CPU= 23.986
  18:31:04 STEP 1330 H= 332:30 +CPU= 23.404
  18:31:31 STEP 1331 H= 332:45 +CPU= 27.303
  18:32:03 STEP 1332 H= 333:00 +CPU= 31.684
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001332
  18:32:26 STEP 1333 H= 333:15 +CPU= 23.504
  18:32:53 STEP 1334 H= 333:30 +CPU= 26.671
  18:33:17 STEP 1335 H= 333:45 +CPU= 24.246
  18:33:43 STEP 1336 H= 334:00 +CPU= 25.726
  18:34:10 STEP 1337 H= 334:15 +CPU= 27.530
  18:34:37 STEP 1338 H= 334:30 +CPU= 26.291
  18:35:03 STEP 1339 H= 334:45 +CPU= 26.246
  18:35:31 STEP 1340 H= 335:00 +CPU= 27.343
  18:35:57 STEP 1341 H= 335:15 +CPU= 26.653
  18:36:23 STEP 1342 H= 335:30 +CPU= 26.232
  18:36:51 STEP 1343 H= 335:45 +CPU= 27.043
  18:37:26 STEP 1344 H= 336:00 +CPU= 35.601
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_13.zip
Uploading the intermediate file: upload_file_13.zip
  18:37:51 STEP 1345 H= 336:15 +CPU= 24.364
  18:38:16 STEP 1346 H= 336:30 +CPU= 25.774
  18:38:40 STEP 1347 H= 336:45 +CPU= 23.749
  18:39:04 STEP 1348 H= 337:00 +CPU= 24.083
  18:39:31 STEP 1349 H= 337:15 +CPU= 26.566
  18:39:57 STEP 1350 H= 337:30 +CPU= 26.055
  18:40:21 STEP 1351 H= 337:45 +CPU= 24.076
  18:40:50 STEP 1352 H= 338:00 +CPU= 28.830
  18:41:13 STEP 1353 H= 338:15 +CPU= 22.658
  18:41:37 STEP 1354 H= 338:30 +CPU= 24.475
  18:42:05 STEP 1355 H= 338:45 +CPU= 27.951
  18:42:38 STEP 1356 H= 339:00 +CPU= 32.586
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001356
  18:43:02 STEP 1357 H= 339:15 +CPU= 24.521
  18:43:32 STEP 1358 H= 339:30 +CPU= 29.177
  18:43:57 STEP 1359 H= 339:45 +CPU= 25.021
  18:44:22 STEP 1360 H= 340:00 +CPU= 25.573
  18:44:49 STEP 1361 H= 340:15 +CPU= 26.814
  18:45:16 STEP 1362 H= 340:30 +CPU= 26.477
  18:45:43 STEP 1363 H= 340:45 +CPU= 27.138
  18:46:09 STEP 1364 H= 341:00 +CPU= 26.256
  18:46:34 STEP 1365 H= 341:15 +CPU= 25.161
  18:46:57 STEP 1366 H= 341:30 +CPU= 23.006
  18:47:24 STEP 1367 H= 341:45 +CPU= 26.625
  18:47:57 STEP 1368 H= 342:00 +CPU= 32.740
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001368
  18:48:24 STEP 1369 H= 342:15 +CPU= 26.907
  18:48:53 STEP 1370 H= 342:30 +CPU= 28.869
  18:49:18 STEP 1371 H= 342:45 +CPU= 25.239
  18:49:44 STEP 1372 H= 343:00 +CPU= 26.456
  18:50:10 STEP 1373 H= 343:15 +CPU= 25.557
  18:50:37 STEP 1374 H= 343:30 +CPU= 26.933
  18:51:02 STEP 1375 H= 343:45 +CPU= 24.937
  18:51:30 STEP 1376 H= 344:00 +CPU= 27.668
  18:51:57 STEP 1377 H= 344:15 +CPU= 26.765
  18:52:22 STEP 1378 H= 344:30 +CPU= 25.212
  18:52:47 STEP 1379 H= 344:45 +CPU= 25.556
  18:53:21 STEP 1380 H= 345:00 +CPU= 33.177
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001380
  18:53:46 STEP 1381 H= 345:15 +CPU= 24.975
  18:54:12 STEP 1382 H= 345:30 +CPU= 26.138
  18:54:39 STEP 1383 H= 345:45 +CPU= 27.149
  18:55:04 STEP 1384 H= 346:00 +CPU= 24.663
  18:55:32 STEP 1385 H= 346:15 +CPU= 27.571
  18:55:56 STEP 1386 H= 346:30 +CPU= 24.574
  18:56:24 STEP 1387 H= 346:45 +CPU= 27.453
  18:56:49 STEP 1388 H= 347:00 +CPU= 25.480
  18:57:17 STEP 1389 H= 347:15 +CPU= 27.857
  18:57:45 STEP 1390 H= 347:30 +CPU= 27.617
  18:58:12 STEP 1391 H= 347:45 +CPU= 27.048
  18:58:44 STEP 1392 H= 348:00 +CPU= 32.283
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001392
  18:59:09 STEP 1393 H= 348:15 +CPU= 25.407
  18:59:36 STEP 1394 H= 348:30 +CPU= 26.290
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.906931107421940E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.801505439615894E-06 0.000000000000000E+00 0.691256509801632E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.928909693762414E-07 0.000000000000000E+00 0.193106172248810E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.907032117535380E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.153011004503388E-06 0.000000000000000E+00 0.419486186343056E-03
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.176524816086415E-06 0.000000000000000E+00 0.338445606571548E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.146081938132930E-06 0.000000000000000E+00 0.360535070063404E-03
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.106891208874304E-07 0.000000000000000E+00 0.204098938681015E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.370909045926569E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1392 STEPO   F00000000
 CREATE RESTART FILE ./srf00141200.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00141200.0001 MODE=w
 IO-STREAM CLOSED - ./srf00141200.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    1.7026 SECONDS******** Mbytes/s, TOTAL TIME=    2.2717(74.9%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00140000.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1392 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001392 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001392
 MPI-TASK:   1 -    313762132 BYTES IN      6 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0072( 2.8%)
 NSTEP =  1392 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001392 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001392
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0048 SECONDS147374.8 Mbytes/s, TOTAL TIME=    0.1976( 2.4%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001392 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001392
 MPI-TASK:   1 -   5020194112 BYTES IN    125 RECORDS TRANSFERRED IN    0.0028 SECONDS******** Mbytes/s, TOTAL TIME=    0.1596( 1.8%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001392 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001392
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0001 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0030( 3.3%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001392 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001392
 MPI-TASK:   1 -    305182403 BYTES IN      4 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0054( 1.9%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1392 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1392 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001392 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001392
 MPI-TASK:   1 -    313762132 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS784405.3 Mbytes/s, TOTAL TIME=    0.0121( 3.3%)
 18:58:44 STEP 1392 H= 348:00 +CPU= 32.283
 NSTEP =  1393 STEPO   0AAA00AAA
 18:59:09 STEP 1393 H= 348:15 +CPU= 25.407
 NSTEP =  1394 STEPO   0AAA00AAA
 18:59:36 STEP 1394 H= 348:30 +CPU= 26.290
ISTEP=    1395 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1395 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1395 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 18:58:16 FULLPOS-B DYNFPOS   1392     1.154    1.154    1.155    508:07    509:40 0.12432687607083E-05       2GB       0MB
 18:58:18 FULLPOS-V DYNFPOS   1392     2.548    2.548    2.548    508:10    509:43 0.12432687607083E-05       2GB       0MB
 18:58:19 FULLPOS-T DYNFPOS   1392     0.463    0.463    0.463    508:10    509:43 0.12432687607083E-05       2GB       0MB
 18:58:19 0AAA00AAA STEPO     1392     0.464    0.464    0.464    508:11    509:43 0.12432687607083E-05       2GB       0MB
 18:58:44 0AAA00AAA STEPO     1393    24.928   24.928   24.937    508:35    510:08 0.12432687607083E-05       2GB       0MB
 18:59:09 0AAA00AAA STEPO     1394    25.407   25.407   25.426    509:01    510:34 0.12432687607083E-05       2GB       0MB
 18:59:36 0AAA00AAA STEPO     1395    26.290   26.290   26.304    509:27    511:00 0.12432687607083E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1392",
 CTIME   = "00141200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1380, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc/slots/7/progress_file_12282797.xml

<running_values>
  <last_cpu_time>40275.700000</last_cpu_time>
  <upload_file_number>14</upload_file_number>
  <last_iter>1395</last_iter>
  <last_upload>1209600</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0qy
(argv4) batchid: 1017
(argv5) wuid: 12282797
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/7
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/7/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_d07bf7fdc240e5d52e8f077cd0445f62 to: /var/lib/boinc/slots/7/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/7/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797.zip
ic_ancil_file: ic_ancil_12282797
ifsdata_file: ifsdata_12282797
climate_data_file: clim_data_12282797
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/7/ic_ancil_12282797.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/7/ic_ancil_12282797.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9b1fb6524de398b8cb8833b7242c94cd to: /var/lib/boinc/slots/7/ifsdata/ifsdata_12282797.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/7/ifsdata/ifsdata_12282797.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/7/159l_2/clim_data_12282797.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/7/159l_2/clim_data_12282797.zip
Checking for progress XML file: /var/lib/boinc/slots/7/progress_file_12282797.xml
Opened progress file ok : /var/lib/boinc/slots/7/progress_file_12282797.xml
-- Model is restarting --
Adjusting last_iter, 1395, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/7/progress_file_12282797.xml
last_cpu_time: 40275
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738
The child process has been launched with process id: 11252
Executing the command: /var/lib/boinc/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:11252:11252] [20240608:210003:1717869603:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:00:33 STEP 1392 H= 348:00 +CPU= 26.041
  21:00:56 STEP 1393 H= 348:15 +CPU= 23.001
  21:01:20 STEP 1394 H= 348:30 +CPU= 24.361
  21:01:45 STEP 1395 H= 348:45 +CPU= 24.085
  21:02:09 STEP 1396 H= 349:00 +CPU= 24.429
  21:02:35 STEP 1397 H= 349:15 +CPU= 26.168
  21:02:59 STEP 1398 H= 349:30 +CPU= 24.065
  21:03:23 STEP 1399 H= 349:45 +CPU= 24.044
  21:03:47 STEP 1400 H= 350:00 +CPU= 23.643
  21:04:12 STEP 1401 H= 350:15 +CPU= 25.018
  21:04:37 STEP 1402 H= 350:30 +CPU= 25.217
  21:05:02 STEP 1403 H= 350:45 +CPU= 24.559
  21:05:34 STEP 1404 H= 351:00 +CPU= 32.462
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001404
  21:05:59 STEP 1405 H= 351:15 +CPU= 24.440
  21:06:24 STEP 1406 H= 351:30 +CPU= 25.122
  21:06:50 STEP 1407 H= 351:45 +CPU= 26.271
  21:07:18 STEP 1408 H= 352:00 +CPU= 27.277
  21:07:43 STEP 1409 H= 352:15 +CPU= 25.393
  21:08:10 STEP 1410 H= 352:30 +CPU= 26.532
  21:08:36 STEP 1411 H= 352:45 +CPU= 26.068
  21:09:01 STEP 1412 H= 353:00 +CPU= 25.650
  21:09:27 STEP 1413 H= 353:15 +CPU= 25.372
  21:09:52 STEP 1414 H= 353:30 +CPU= 24.906
  21:10:17 STEP 1415 H= 353:45 +CPU= 25.318
  21:10:47 STEP 1416 H= 354:00 +CPU= 30.172
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001416
  21:11:12 STEP 1417 H= 354:15 +CPU= 24.525
  21:11:36 STEP 1418 H= 354:30 +CPU= 24.299
  21:12:00 STEP 1419 H= 354:45 +CPU= 24.254
  21:12:24 STEP 1420 H= 355:00 +CPU= 23.236
  21:12:45 STEP 1421 H= 355:15 +CPU= 21.249
  21:13:07 STEP 1422 H= 355:30 +CPU= 22.520
  21:13:31 STEP 1423 H= 355:45 +CPU= 23.916
  21:13:54 STEP 1424 H= 356:00 +CPU= 23.149
  21:14:17 STEP 1425 H= 356:15 +CPU= 22.169
  21:14:42 STEP 1426 H= 356:30 +CPU= 25.579
  21:15:05 STEP 1427 H= 356:45 +CPU= 22.244
  21:15:33 STEP 1428 H= 357:00 +CPU= 27.999
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001428
  21:15:57 STEP 1429 H= 357:15 +CPU= 24.585
  21:16:20 STEP 1430 H= 357:30 +CPU= 23.185
  21:16:44 STEP 1431 H= 357:45 +CPU= 23.624
  21:17:07 STEP 1432 H= 358:00 +CPU= 22.774
  21:17:29 STEP 1433 H= 358:15 +CPU= 22.547
  21:17:52 STEP 1434 H= 358:30 +CPU= 22.205
  21:18:14 STEP 1435 H= 358:45 +CPU= 22.635
  21:18:36 STEP 1436 H= 359:00 +CPU= 21.935
  21:18:59 STEP 1437 H= 359:15 +CPU= 22.488
  21:19:21 STEP 1438 H= 359:30 +CPU= 21.909
  21:19:42 STEP 1439 H= 359:45 +CPU= 21.658
  21:20:11 STEP 1440 H= 360:00 +CPU= 28.566
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1012739201E-20      I       0.3775773652E-05      I
ICloud fractionI      0.0000000000E+00      I       0.2223987390E-01      I
I  Relat. Hum. I      0.2515303574E+00      I       0.3214674447E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3070163313E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    325173588 BYTES IN      7 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0077( 2.6%)
 21:20:11 STEP 1440 H= 360:00 +CPU= 28.566
 END CNT3
 NSTEP =  1440 CNT0   000000000  21.325  21.325  21.474   0.993               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************     3.0     1.6     1.4     2.3     3.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1207463.7 1207463.7    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1203892.4 1203892.4     99.70      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    3638.9    3638.9     15.97      0.00
   9 SPCM     - SPECTRAL COMP.                      48     682.4     682.4      2.71      0.00
  10 SCAN2M - PHYSICS                               49   12337.5   12337.5     50.07      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     537.8     537.8      0.18      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      38.7      38.7      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      19.4      19.4      0.01      0.00
  27 RESTART FILES - WRITING                         1    2151.1    2151.1      0.18      0.00
  28 RESTART FILES - READING                         1    3023.2    3023.2      0.25      0.00
  29 SU4FPOS IN CNT4                                 4       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    5329.7    5329.7      1.77      0.00
  31 POSDDH IN STEPO                                 1       0.8       0.8      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    4932.0    4932.0     20.01      0.00
  39 SU0YOMB                                         1     372.3     372.3      0.03      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      25.6      25.6      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     311.1     311.1      1.24      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      43.9      43.9      0.17      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     336.6     336.6      3.01      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     206.0     206.0      1.71      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      72.3      72.3      0.60      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108     116.5     116.5      1.04      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      32.4      32.4      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      39.4      39.4      0.35      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      22.3      22.3      0.18      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      67.4      67.4      0.60      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      18.9      18.9      0.16      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1207.5 CPU TIME    1206.4 VECTOR TIME     1206.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1076.5
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/7/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/7/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/7/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/7/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/7/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282797/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
21:23:40 (10920): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0_r568013738_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 18:21:59 1472385 22434366 oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0 1,295,100 41,481 0.0320
08 Jun 2024 14:57:12 1472385 22434366 oifs_43r3_bl_a0qy_2016092300_20_1017_12282797_0 1,123,200 36,403 0.0324


©2024 cpdn.org