climateprediction.net home page
Task 22434163

Task 22434163

Name oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0
Workunit 12282594
Created 7 Jun 2024, 11:57:25 UTC
Sent 7 Jun 2024, 13:20:49 UTC
Report deadline 6 Aug 2024, 13:20:49 UTC
Received 10 Jun 2024, 12:15:49 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1509254
Run time 11 hours 47 min 49 sec
CPU time 11 hours 47 min 49 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 4.49 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,023.90 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,367.45 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
00 +CPU= 28.334
  19:09:42 STEP 1061 H= 265:15 +CPU= 28.431
  19:10:09 STEP 1062 H= 265:30 +CPU= 27.642
  19:10:38 STEP 1063 H= 265:45 +CPU= 28.784
  19:11:07 STEP 1064 H= 266:00 +CPU= 29.083
  19:11:36 STEP 1065 H= 266:15 +CPU= 28.294
  19:12:04 STEP 1066 H= 266:30 +CPU= 28.784
  19:12:33 STEP 1067 H= 266:45 +CPU= 28.435
  19:13:10 STEP 1068 H= 267:00 +CPU= 37.558
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001068
  19:13:39 STEP 1069 H= 267:15 +CPU= 28.338
  19:14:06 STEP 1070 H= 267:30 +CPU= 27.185
  19:14:34 STEP 1071 H= 267:45 +CPU= 27.979
  19:15:04 STEP 1072 H= 268:00 +CPU= 29.492
  19:15:31 STEP 1073 H= 268:15 +CPU= 27.117
  19:15:57 STEP 1074 H= 268:30 +CPU= 26.196
  19:16:25 STEP 1075 H= 268:45 +CPU= 28.285
  19:16:53 STEP 1076 H= 269:00 +CPU= 27.328
  19:17:23 STEP 1077 H= 269:15 +CPU= 29.884
  19:17:51 STEP 1078 H= 269:30 +CPU= 28.355
  19:18:17 STEP 1079 H= 269:45 +CPU= 26.026
  19:18:53 STEP 1080 H= 270:00 +CPU= 35.707
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001080
  19:19:21 STEP 1081 H= 270:15 +CPU= 28.120
  19:19:48 STEP 1082 H= 270:30 +CPU= 27.362
  19:20:16 STEP 1083 H= 270:45 +CPU= 28.102
  19:20:44 STEP 1084 H= 271:00 +CPU= 27.415
  19:21:11 STEP 1085 H= 271:15 +CPU= 26.672
  19:21:39 STEP 1086 H= 271:30 +CPU= 28.838
  19:22:06 STEP 1087 H= 271:45 +CPU= 26.424
  19:22:35 STEP 1088 H= 272:00 +CPU= 29.059
  19:23:05 STEP 1089 H= 272:15 +CPU= 29.524
  19:23:33 STEP 1090 H= 272:30 +CPU= 28.679
  19:24:03 STEP 1091 H= 272:45 +CPU= 29.321
  19:24:38 STEP 1092 H= 273:00 +CPU= 35.710
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001092
  19:25:08 STEP 1093 H= 273:15 +CPU= 29.380
  19:25:37 STEP 1094 H= 273:30 +CPU= 29.552
  19:26:06 STEP 1095 H= 273:45 +CPU= 28.696
  19:26:35 STEP 1096 H= 274:00 +CPU= 29.113
  19:27:05 STEP 1097 H= 274:15 +CPU= 29.314
  19:27:31 STEP 1098 H= 274:30 +CPU= 26.734
  19:27:59 STEP 1099 H= 274:45 +CPU= 27.778
  19:28:29 STEP 1100 H= 275:00 +CPU= 29.441
  19:28:57 STEP 1101 H= 275:15 +CPU= 28.754
  19:29:26 STEP 1102 H= 275:30 +CPU= 28.080
  19:29:53 STEP 1103 H= 275:45 +CPU= 27.432
  19:30:31 STEP 1104 H= 276:00 +CPU= 38.172
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001104
  19:30:58 STEP 1105 H= 276:15 +CPU= 27.027
  19:31:27 STEP 1106 H= 276:30 +CPU= 28.472
  19:31:56 STEP 1107 H= 276:45 +CPU= 29.146
  19:32:24 STEP 1108 H= 277:00 +CPU= 27.819
  19:32:52 STEP 1109 H= 277:15 +CPU= 27.869
  19:33:22 STEP 1110 H= 277:30 +CPU= 30.228
  19:33:51 STEP 1111 H= 277:45 +CPU= 28.507
  19:34:18 STEP 1112 H= 278:00 +CPU= 27.347
  19:34:45 STEP 1113 H= 278:15 +CPU= 26.807
  19:35:14 STEP 1114 H= 278:30 +CPU= 29.395
  19:35:44 STEP 1115 H= 278:45 +CPU= 29.475
  19:36:17 STEP 1116 H= 279:00 +CPU= 33.381
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001116
  19:36:45 STEP 1117 H= 279:15 +CPU= 28.172
  19:37:12 STEP 1118 H= 279:30 +CPU= 26.954
  19:37:39 STEP 1119 H= 279:45 +CPU= 26.681
  19:38:07 STEP 1120 H= 280:00 +CPU= 27.630
  19:38:35 STEP 1121 H= 280:15 +CPU= 28.392
  19:39:04 STEP 1122 H= 280:30 +CPU= 29.303
  19:39:33 STEP 1123 H= 280:45 +CPU= 28.277
  19:40:02 STEP 1124 H= 281:00 +CPU= 29.010
  19:40:30 STEP 1125 H= 281:15 +CPU= 28.064
  19:40:58 STEP 1126 H= 281:30 +CPU= 27.884
  19:41:25 STEP 1127 H= 281:45 +CPU= 27.186
  19:41:59 STEP 1128 H= 282:00 +CPU= 33.981
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001128
  19:42:29 STEP 1129 H= 282:15 +CPU= 29.327
  19:42:57 STEP 1130 H= 282:30 +CPU= 28.111
  19:43:26 STEP 1131 H= 282:45 +CPU= 28.973
  19:43:53 STEP 1132 H= 283:00 +CPU= 27.293
  19:44:22 STEP 1133 H= 283:15 +CPU= 28.562
  19:44:51 STEP 1134 H= 283:30 +CPU= 29.293
  19:45:20 STEP 1135 H= 283:45 +CPU= 28.823
  19:45:48 STEP 1136 H= 284:00 +CPU= 28.651
  19:46:16 STEP 1137 H= 284:15 +CPU= 27.280
  19:46:44 STEP 1138 H= 284:30 +CPU= 28.487
  19:47:12 STEP 1139 H= 284:45 +CPU= 28.008
  19:47:50 STEP 1140 H= 285:00 +CPU= 37.787
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001140
  19:48:18 STEP 1141 H= 285:15 +CPU= 27.726
  19:48:46 STEP 1142 H= 285:30 +CPU= 28.161
  19:49:15 STEP 1143 H= 285:45 +CPU= 28.634
  19:49:45 STEP 1144 H= 286:00 +CPU= 30.218
  19:50:13 STEP 1145 H= 286:15 +CPU= 27.777
  19:50:43 STEP 1146 H= 286:30 +CPU= 29.941
  19:51:12 STEP 1147 H= 286:45 +CPU= 29.272
  19:51:42 STEP 1148 H= 287:00 +CPU= 29.553
  19:52:12 STEP 1149 H= 287:15 +CPU= 30.022
  19:52:41 STEP 1150 H= 287:30 +CPU= 28.946
  19:53:10 STEP 1151 H= 287:45 +CPU= 29.221
  19:53:47 STEP 1152 H= 288:00 +CPU= 37.202
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_11.zip
Uploading the intermediate file: upload_file_11.zip
  19:54:16 STEP 1153 H= 288:15 +CPU= 28.401
  19:54:46 STEP 1154 H= 288:30 +CPU= 29.970
  19:55:13 STEP 1155 H= 288:45 +CPU= 27.496
  19:55:45 STEP 1156 H= 289:00 +CPU= 31.998
  19:56:14 STEP 1157 H= 289:15 +CPU= 28.247
  19:56:41 STEP 1158 H= 289:30 +CPU= 27.408
  19:57:09 STEP 1159 H= 289:45 +CPU= 27.686
  19:57:38 STEP 1160 H= 290:00 +CPU= 29.343
  19:58:10 STEP 1161 H= 290:15 +CPU= 31.935
  19:58:40 STEP 1162 H= 290:30 +CPU= 29.632
  19:59:08 STEP 1163 H= 290:45 +CPU= 28.338
  19:59:42 STEP 1164 H= 291:00 +CPU= 33.457
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001164
  20:00:09 STEP 1165 H= 291:15 +CPU= 27.700
  20:00:37 STEP 1166 H= 291:30 +CPU= 27.757
  20:01:07 STEP 1167 H= 291:45 +CPU= 29.319
  20:01:35 STEP 1168 H= 292:00 +CPU= 28.188
  20:02:02 STEP 1169 H= 292:15 +CPU= 26.878
  20:02:31 STEP 1170 H= 292:30 +CPU= 29.467
  20:03:00 STEP 1171 H= 292:45 +CPU= 29.197
  20:03:29 STEP 1172 H= 293:00 +CPU= 28.634
  20:03:58 STEP 1173 H= 293:15 +CPU= 29.023
  20:04:25 STEP 1174 H= 293:30 +CPU= 27.374
  20:04:54 STEP 1175 H= 293:45 +CPU= 28.171
  20:05:27 STEP 1176 H= 294:00 +CPU= 33.362
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001176
  20:05:56 STEP 1177 H= 294:15 +CPU= 28.595
  20:06:23 STEP 1178 H= 294:30 +CPU= 27.249
  20:06:52 STEP 1179 H= 294:45 +CPU= 28.498
  20:07:21 STEP 1180 H= 295:00 +CPU= 29.793
  20:07:51 STEP 1181 H= 295:15 +CPU= 29.261
  20:08:19 STEP 1182 H= 295:30 +CPU= 28.572
  20:08:49 STEP 1183 H= 295:45 +CPU= 29.284
  20:09:17 STEP 1184 H= 296:00 +CPU= 28.401
  20:09:46 STEP 1185 H= 296:15 +CPU= 29.185
  20:10:16 STEP 1186 H= 296:30 +CPU= 30.190
  20:10:46 STEP 1187 H= 296:45 +CPU= 29.526
  20:11:23 STEP 1188 H= 297:00 +CPU= 36.928
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001188
  20:11:51 STEP 1189 H= 297:15 +CPU= 28.424
  20:12:21 STEP 1190 H= 297:30 +CPU= 29.824
  20:12:52 STEP 1191 H= 297:45 +CPU= 30.500
  20:13:22 STEP 1192 H= 298:00 +CPU= 29.841
  20:13:50 STEP 1193 H= 298:15 +CPU= 28.716
  20:14:19 STEP 1194 H= 298:30 +CPU= 28.619
  20:14:49 STEP 1195 H= 298:45 +CPU= 29.757
  20:15:17 STEP 1196 H= 299:00 +CPU= 28.241
  20:15:45 STEP 1197 H= 299:15 +CPU= 28.320
  20:16:14 STEP 1198 H= 299:30 +CPU= 28.661
  20:16:42 STEP 1199 H= 299:45 +CPU= 27.960
  20:17:19 STEP 1200 H= 300:00 +CPU= 37.092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001200
  20:17:47 STEP 1201 H= 300:15 +CPU= 27.934
  20:18:15 STEP 1202 H= 300:30 +CPU= 27.477
  20:18:42 STEP 1203 H= 300:45 +CPU= 27.380
  20:19:10 STEP 1204 H= 301:00 +CPU= 27.534
  20:19:39 STEP 1205 H= 301:15 +CPU= 29.240
  20:20:07 STEP 1206 H= 301:30 +CPU= 27.625
  20:20:35 STEP 1207 H= 301:45 +CPU= 28.408
  20:21:04 STEP 1208 H= 302:00 +CPU= 29.242
  20:21:31 STEP 1209 H= 302:15 +CPU= 26.396
  20:21:59 STEP 1210 H= 302:30 +CPU= 28.530
  20:22:28 STEP 1211 H= 302:45 +CPU= 28.247
  20:23:03 STEP 1212 H= 303:00 +CPU= 34.830
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001212
  20:23:31 STEP 1213 H= 303:15 +CPU= 28.136
  20:23:58 STEP 1214 H= 303:30 +CPU= 27.679
  20:24:30 STEP 1215 H= 303:45 +CPU= 31.365
  20:25:01 STEP 1216 H= 304:00 +CPU= 31.383
  20:25:29 STEP 1217 H= 304:15 +CPU= 27.547
  20:25:58 STEP 1218 H= 304:30 +CPU= 29.290
  20:26:27 STEP 1219 H= 304:45 +CPU= 28.883
  20:26:58 STEP 1220 H= 305:00 +CPU= 30.932
  20:27:25 STEP 1221 H= 305:15 +CPU= 27.202
  20:27:53 STEP 1222 H= 305:30 +CPU= 27.938
  20:28:21 STEP 1223 H= 305:45 +CPU= 28.240
  20:28:58 STEP 1224 H= 306:00 +CPU= 36.073
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001224
  20:29:26 STEP 1225 H= 306:15 +CPU= 28.413
  20:29:54 STEP 1226 H= 306:30 +CPU= 28.127
  20:30:25 STEP 1227 H= 306:45 +CPU= 30.639
  20:30:54 STEP 1228 H= 307:00 +CPU= 29.228
  20:31:21 STEP 1229 H= 307:15 +CPU= 26.931
  20:31:47 STEP 1230 H= 307:30 +CPU= 26.028
  20:32:17 STEP 1231 H= 307:45 +CPU= 30.056
  20:32:47 STEP 1232 H= 308:00 +CPU= 29.780
  20:33:15 STEP 1233 H= 308:15 +CPU= 27.621
  20:33:43 STEP 1234 H= 308:30 +CPU= 27.900
  20:34:13 STEP 1235 H= 308:45 +CPU= 30.679
  20:34:47 STEP 1236 H= 309:00 +CPU= 34.178
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001236
  20:35:19 STEP 1237 H= 309:15 +CPU= 31.267
  20:35:49 STEP 1238 H= 309:30 +CPU= 30.335
  20:36:18 STEP 1239 H= 309:45 +CPU= 28.484
  20:36:47 STEP 1240 H= 310:00 +CPU= 29.730
  20:37:16 STEP 1241 H= 310:15 +CPU= 28.968
  20:37:46 STEP 1242 H= 310:30 +CPU= 29.156
  20:38:15 STEP 1243 H= 310:45 +CPU= 29.153
  20:38:45 STEP 1244 H= 311:00 +CPU= 29.960
  20:39:14 STEP 1245 H= 311:15 +CPU= 29.150
  20:39:44 STEP 1246 H= 311:30 +CPU= 29.995
  20:40:11 STEP 1247 H= 311:45 +CPU= 27.029
  20:40:48 STEP 1248 H= 312:00 +CPU= 37.154
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_12.zip
Uploading the intermediate file: upload_file_12.zip
  20:41:17 STEP 1249 H= 312:15 +CPU= 29.175
  20:41:47 STEP 1250 H= 312:30 +CPU= 29.119
  20:42:16 STEP 1251 H= 312:45 +CPU= 29.691
  20:42:47 STEP 1252 H= 313:00 +CPU= 30.790
  20:43:16 STEP 1253 H= 313:15 +CPU= 28.637
  20:43:45 STEP 1254 H= 313:30 +CPU= 29.259
  20:44:15 STEP 1255 H= 313:45 +CPU= 30.078
  20:44:42 STEP 1256 H= 314:00 +CPU= 26.591
  20:45:11 STEP 1257 H= 314:15 +CPU= 29.623
  20:45:40 STEP 1258 H= 314:30 +CPU= 28.783
  20:46:10 STEP 1259 H= 314:45 +CPU= 30.019
  20:46:46 STEP 1260 H= 315:00 +CPU= 35.879
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001260
  20:47:16 STEP 1261 H= 315:15 +CPU= 29.985
  20:47:45 STEP 1262 H= 315:30 +CPU= 28.741
  20:48:15 STEP 1263 H= 315:45 +CPU= 29.849
  20:48:45 STEP 1264 H= 316:00 +CPU= 29.740
  20:49:13 STEP 1265 H= 316:15 +CPU= 28.687
  20:49:43 STEP 1266 H= 316:30 +CPU= 29.259
  20:50:10 STEP 1267 H= 316:45 +CPU= 27.175
  20:50:40 STEP 1268 H= 317:00 +CPU= 29.687
  20:51:07 STEP 1269 H= 317:15 +CPU= 27.363
  20:51:35 STEP 1270 H= 317:30 +CPU= 28.274
  20:52:06 STEP 1271 H= 317:45 +CPU= 30.209
  20:52:41 STEP 1272 H= 318:00 +CPU= 35.046
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001272
  20:53:12 STEP 1273 H= 318:15 +CPU= 31.067
  20:53:42 STEP 1274 H= 318:30 +CPU= 30.720
  20:54:11 STEP 1275 H= 318:45 +CPU= 28.442
  20:54:39 STEP 1276 H= 319:00 +CPU= 28.168
  20:55:08 STEP 1277 H= 319:15 +CPU= 28.728
  20:55:37 STEP 1278 H= 319:30 +CPU= 29.591
  20:56:08 STEP 1279 H= 319:45 +CPU= 30.310
  20:56:38 STEP 1280 H= 320:00 +CPU= 30.254
  20:57:06 STEP 1281 H= 320:15 +CPU= 28.129
  20:57:34 STEP 1282 H= 320:30 +CPU= 28.161
  20:58:01 STEP 1283 H= 320:45 +CPU= 26.701
  20:58:37 STEP 1284 H= 321:00 +CPU= 35.942
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001284
  20:59:07 STEP 1285 H= 321:15 +CPU= 29.695
  20:59:35 STEP 1286 H= 321:30 +CPU= 28.307
  21:00:04 STEP 1287 H= 321:45 +CPU= 28.572
  21:00:33 STEP 1288 H= 322:00 +CPU= 29.637
  21:01:01 STEP 1289 H= 322:15 +CPU= 28.025
  21:01:29 STEP 1290 H= 322:30 +CPU= 27.911
  21:01:58 STEP 1291 H= 322:45 +CPU= 28.512
  21:02:28 STEP 1292 H= 323:00 +CPU= 29.621
  21:02:54 STEP 1293 H= 323:15 +CPU= 26.213
  21:03:24 STEP 1294 H= 323:30 +CPU= 30.632
  21:03:53 STEP 1295 H= 323:45 +CPU= 28.797
  21:04:33 STEP 1296 H= 324:00 +CPU= 38.880
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001296
  21:05:02 STEP 1297 H= 324:15 +CPU= 29.164
  21:05:29 STEP 1298 H= 324:30 +CPU= 27.682
  21:05:58 STEP 1299 H= 324:45 +CPU= 28.272
  21:06:29 STEP 1300 H= 325:00 +CPU= 31.676
  21:06:57 STEP 1301 H= 325:15 +CPU= 28.003
  21:07:27 STEP 1302 H= 325:30 +CPU= 29.649
  21:07:56 STEP 1303 H= 325:45 +CPU= 28.509
  21:08:26 STEP 1304 H= 326:00 +CPU= 30.114
  21:08:55 STEP 1305 H= 326:15 +CPU= 29.430
  21:09:24 STEP 1306 H= 326:30 +CPU= 28.271
  21:09:53 STEP 1307 H= 326:45 +CPU= 28.970
  21:10:28 STEP 1308 H= 327:00 +CPU= 35.604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001308
  21:10:57 STEP 1309 H= 327:15 +CPU= 28.398
  21:11:25 STEP 1310 H= 327:30 +CPU= 28.044
  21:11:54 STEP 1311 H= 327:45 +CPU= 29.060
  21:12:21 STEP 1312 H= 328:00 +CPU= 27.089
  21:12:50 STEP 1313 H= 328:15 +CPU= 29.364
  21:13:20 STEP 1314 H= 328:30 +CPU= 29.345
  21:13:51 STEP 1315 H= 328:45 +CPU= 30.924
  21:14:20 STEP 1316 H= 329:00 +CPU= 29.100
  21:14:49 STEP 1317 H= 329:15 +CPU= 29.207
  21:15:18 STEP 1318 H= 329:30 +CPU= 29.350
  21:15:48 STEP 1319 H= 329:45 +CPU= 29.077
  21:16:21 STEP 1320 H= 330:00 +CPU= 33.934
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001320
  21:16:51 STEP 1321 H= 330:15 +CPU= 29.887
  21:17:23 STEP 1322 H= 330:30 +CPU= 31.376
  21:17:51 STEP 1323 H= 330:45 +CPU= 27.976
  21:18:20 STEP 1324 H= 331:00 +CPU= 28.984
  21:18:50 STEP 1325 H= 331:15 +CPU= 29.904
  21:19:17 STEP 1326 H= 331:30 +CPU= 27.533
  21:19:46 STEP 1327 H= 331:45 +CPU= 28.559
  21:20:15 STEP 1328 H= 332:00 +CPU= 28.834
  21:20:43 STEP 1329 H= 332:15 +CPU= 27.786
  21:21:12 STEP 1330 H= 332:30 +CPU= 29.414
  21:21:40 STEP 1331 H= 332:45 +CPU= 28.494
  21:22:16 STEP 1332 H= 333:00 +CPU= 35.126
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001332
  21:22:45 STEP 1333 H= 333:15 +CPU= 28.985
  21:23:14 STEP 1334 H= 333:30 +CPU= 29.195
  21:23:43 STEP 1335 H= 333:45 +CPU= 28.725
  21:24:12 STEP 1336 H= 334:00 +CPU= 29.469
  21:24:43 STEP 1337 H= 334:15 +CPU= 30.572
  21:25:12 STEP 1338 H= 334:30 +CPU= 29.096
  21:25:40 STEP 1339 H= 334:45 +CPU= 28.386
  21:26:10 STEP 1340 H= 335:00 +CPU= 30.055
  21:26:40 STEP 1341 H= 335:15 +CPU= 29.499
  21:27:08 STEP 1342 H= 335:30 +CPU= 28.479
  21:27:39 STEP 1343 H= 335:45 +CPU= 30.392
  21:28:17 STEP 1344 H= 336:00 +CPU= 38.175
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_13.zip
Uploading the intermediate file: upload_file_13.zip
  21:28:46 STEP 1345 H= 336:15 +CPU= 28.466
  21:29:14 STEP 1346 H= 336:30 +CPU= 27.890
  21:29:44 STEP 1347 H= 336:45 +CPU= 30.421
  21:30:13 STEP 1348 H= 337:00 +CPU= 28.858
  21:30:43 STEP 1349 H= 337:15 +CPU= 30.514
  21:31:13 STEP 1350 H= 337:30 +CPU= 29.674
  21:31:43 STEP 1351 H= 337:45 +CPU= 29.597
  21:32:13 STEP 1352 H= 338:00 +CPU= 30.179
  21:32:42 STEP 1353 H= 338:15 +CPU= 29.255
  21:33:11 STEP 1354 H= 338:30 +CPU= 28.668
  21:33:40 STEP 1355 H= 338:45 +CPU= 28.810
  21:34:14 STEP 1356 H= 339:00 +CPU= 33.782
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001356
  21:34:43 STEP 1357 H= 339:15 +CPU= 29.281
  21:35:11 STEP 1358 H= 339:30 +CPU= 28.472
  21:35:41 STEP 1359 H= 339:45 +CPU= 29.897
  21:36:11 STEP 1360 H= 340:00 +CPU= 30.053
  21:36:39 STEP 1361 H= 340:15 +CPU= 27.487
  21:37:08 STEP 1362 H= 340:30 +CPU= 28.606
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0lb
(argv4) batchid: 1017
(argv5) wuid: 12282594
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9b0ff390413f1c2616bd7b3a930a311c to: /var/lib/boinc-client/slots/6/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594.zip
ic_ancil_file: ic_ancil_12282594
ifsdata_file: ifsdata_12282594
climate_data_file: clim_data_12282594
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/6/ic_ancil_12282594.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12282594.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fb796b20d26cd0ad550edd8140ca3a83 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12282594.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12282594.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12282594.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12282594.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12282594.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12282594.xml
-- Model is restarting --
Adjusting last_iter, 1363, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12282594.xml
last_cpu_time: 39056
upload_file_number: 14
last_iter: 1345
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791
The child process has been launched with process id: 764689
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:764689:764689] [20240610:053537:1718012137:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:36:36 STEP 1344 H= 336:00 +CPU= 35.622
  05:37:11 STEP 1345 H= 336:15 +CPU= 35.022
  05:37:49 STEP 1346 H= 336:30 +CPU= 36.685
  05:38:23 STEP 1347 H= 336:45 +CPU= 34.265
  05:39:00 STEP 1348 H= 337:00 +CPU= 36.476
  05:39:34 STEP 1349 H= 337:15 +CPU= 34.075
  05:40:07 STEP 1350 H= 337:30 +CPU= 33.595
  05:41:06 STEP 1351 H= 337:45 +CPU= 39.868
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0lb
(argv4) batchid: 1017
(argv5) wuid: 12282594
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9b0ff390413f1c2616bd7b3a930a311c to: /var/lib/boinc-client/slots/6/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594.zip
ic_ancil_file: ic_ancil_12282594
ifsdata_file: ifsdata_12282594
climate_data_file: clim_data_12282594
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/6/ic_ancil_12282594.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12282594.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fb796b20d26cd0ad550edd8140ca3a83 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12282594.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12282594.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12282594.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12282594.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12282594.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12282594.xml
-- Model is restarting --
Adjusting last_iter, 1352, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12282594.xml
last_cpu_time: 39346
upload_file_number: 14
last_iter: 1345
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791
The child process has been launched with process id: 773572
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:HP-Z420-WorkStation1:1:1:773572:773572] [20240610:070631:1718017591:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:07:08 STEP 1344 H= 336:00 +CPU= 34.050
  07:07:44 STEP 1345 H= 336:15 +CPU= 35.126
  07:08:20 STEP 1346 H= 336:30 +CPU= 36.324
  07:08:55 STEP 1347 H= 336:45 +CPU= 34.536
  07:09:31 STEP 1348 H= 337:00 +CPU= 36.508
  07:10:05 STEP 1349 H= 337:15 +CPU= 34.008
  07:10:42 STEP 1350 H= 337:30 +CPU= 36.398
  07:11:17 STEP 1351 H= 337:45 +CPU= 34.858
  07:11:53 STEP 1352 H= 338:00 +CPU= 36.157
  07:12:26 STEP 1353 H= 338:15 +CPU= 32.229
  07:13:00 STEP 1354 H= 338:30 +CPU= 34.337
  07:13:36 STEP 1355 H= 338:45 +CPU= 35.410
  07:14:21 STEP 1356 H= 339:00 +CPU= 45.192
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001356
  07:14:55 STEP 1357 H= 339:15 +CPU= 33.704
  07:15:28 STEP 1358 H= 339:30 +CPU= 32.909
  07:16:01 STEP 1359 H= 339:45 +CPU= 33.494
  07:16:37 STEP 1360 H= 340:00 +CPU= 35.998
  07:17:11 STEP 1361 H= 340:15 +CPU= 33.356
  07:17:47 STEP 1362 H= 340:30 +CPU= 35.527
  07:18:20 STEP 1363 H= 340:45 +CPU= 33.794
  07:18:56 STEP 1364 H= 341:00 +CPU= 35.353
  07:19:30 STEP 1365 H= 341:15 +CPU= 34.331
  07:20:04 STEP 1366 H= 341:30 +CPU= 33.241
  07:20:37 STEP 1367 H= 341:45 +CPU= 32.956
  07:21:18 STEP 1368 H= 342:00 +CPU= 41.380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001368
  07:21:52 STEP 1369 H= 342:15 +CPU= 33.717
  07:22:25 STEP 1370 H= 342:30 +CPU= 32.517
  07:22:58 STEP 1371 H= 342:45 +CPU= 33.707
  07:23:32 STEP 1372 H= 343:00 +CPU= 33.400
  07:24:07 STEP 1373 H= 343:15 +CPU= 34.472
  07:24:38 STEP 1374 H= 343:30 +CPU= 31.665
  07:25:11 STEP 1375 H= 343:45 +CPU= 32.606
  07:25:48 STEP 1376 H= 344:00 +CPU= 36.497
  07:26:23 STEP 1377 H= 344:15 +CPU= 35.266
  07:26:58 STEP 1378 H= 344:30 +CPU= 34.559
  07:27:32 STEP 1379 H= 344:45 +CPU= 34.689
  07:28:16 STEP 1380 H= 345:00 +CPU= 43.869
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001380
  07:28:52 STEP 1381 H= 345:15 +CPU= 35.365
  07:29:27 STEP 1382 H= 345:30 +CPU= 34.946
  07:30:02 STEP 1383 H= 345:45 +CPU= 34.746
  07:30:36 STEP 1384 H= 346:00 +CPU= 33.931
  07:31:09 STEP 1385 H= 346:15 +CPU= 33.350
  07:31:45 STEP 1386 H= 346:30 +CPU= 35.253
  07:32:20 STEP 1387 H= 346:45 +CPU= 35.365
  07:32:53 STEP 1388 H= 347:00 +CPU= 32.372
  07:33:26 STEP 1389 H= 347:15 +CPU= 33.446
  07:33:55 STEP 1390 H= 347:30 +CPU= 28.602
  07:34:24 STEP 1391 H= 347:45 +CPU= 29.239
  07:35:05 STEP 1392 H= 348:00 +CPU= 40.938
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001392
  07:35:35 STEP 1393 H= 348:15 +CPU= 29.304
  07:36:01 STEP 1394 H= 348:30 +CPU= 25.704
  07:36:30 STEP 1395 H= 348:45 +CPU= 29.288
  07:36:58 STEP 1396 H= 349:00 +CPU= 28.234
  07:37:25 STEP 1397 H= 349:15 +CPU= 26.553
  07:37:54 STEP 1398 H= 349:30 +CPU= 29.140
  07:38:19 STEP 1399 H= 349:45 +CPU= 25.117
  07:38:47 STEP 1400 H= 350:00 +CPU= 28.007
  07:39:15 STEP 1401 H= 350:15 +CPU= 27.809
  07:39:43 STEP 1402 H= 350:30 +CPU= 28.360
  07:40:12 STEP 1403 H= 350:45 +CPU= 28.115
  07:40:49 STEP 1404 H= 351:00 +CPU= 36.916
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001404
  07:41:17 STEP 1405 H= 351:15 +CPU= 28.700
  07:41:45 STEP 1406 H= 351:30 +CPU= 27.312
  07:42:11 STEP 1407 H= 351:45 +CPU= 26.545
  07:42:41 STEP 1408 H= 352:00 +CPU= 29.968
  07:43:10 STEP 1409 H= 352:15 +CPU= 28.639
  07:43:41 STEP 1410 H= 352:30 +CPU= 30.474
  07:44:09 STEP 1411 H= 352:45 +CPU= 28.528
  07:44:38 STEP 1412 H= 353:00 +CPU= 28.570
  07:45:07 STEP 1413 H= 353:15 +CPU= 29.263
  07:45:36 STEP 1414 H= 353:30 +CPU= 28.767
  07:46:05 STEP 1415 H= 353:45 +CPU= 28.845
  07:46:41 STEP 1416 H= 354:00 +CPU= 36.041
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001416
  07:47:08 STEP 1417 H= 354:15 +CPU= 27.343
  07:47:39 STEP 1418 H= 354:30 +CPU= 30.265
  07:48:08 STEP 1419 H= 354:45 +CPU= 29.509
  07:48:38 STEP 1420 H= 355:00 +CPU= 29.236
  07:49:05 STEP 1421 H= 355:15 +CPU= 27.623
  07:49:31 STEP 1422 H= 355:30 +CPU= 25.594
  07:50:03 STEP 1423 H= 355:45 +CPU= 32.115
  07:50:33 STEP 1424 H= 356:00 +CPU= 29.781
  07:51:02 STEP 1425 H= 356:15 +CPU= 29.313
  07:51:32 STEP 1426 H= 356:30 +CPU= 29.770
  07:52:03 STEP 1427 H= 356:45 +CPU= 30.811
  07:52:39 STEP 1428 H= 357:00 +CPU= 36.147
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001428
  07:53:09 STEP 1429 H= 357:15 +CPU= 29.287
  07:53:36 STEP 1430 H= 357:30 +CPU= 26.878
  07:54:03 STEP 1431 H= 357:45 +CPU= 27.706
  07:54:36 STEP 1432 H= 358:00 +CPU= 32.283
  07:55:07 STEP 1433 H= 358:15 +CPU= 30.979
  07:55:37 STEP 1434 H= 358:30 +CPU= 30.048
  07:56:06 STEP 1435 H= 358:45 +CPU= 29.515
  07:56:35 STEP 1436 H= 359:00 +CPU= 28.972
  07:57:04 STEP 1437 H= 359:15 +CPU= 28.554
  07:57:32 STEP 1438 H= 359:30 +CPU= 27.792
  07:58:04 STEP 1439 H= 359:45 +CPU= 31.980
  07:58:43 STEP 1440 H= 360:00 +CPU= 39.027
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I      0.3416587744E-22      I       0.8248550498E-06      I
ICloud fractionI      0.0000000000E+00      I       0.3609821493E-01      I
I  Relat. Hum. I      0.9646826573E-01      I       0.2344590493E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.3252954680E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    322837812 BYTES IN      7 RECORDS TRANSFERRED IN    0.0005 SECONDS645675.6 Mbytes/s, TOTAL TIME=    0.0146( 3.4%)
 07:58:43 STEP 1440 H= 360:00 +CPU= 39.027
 END CNT3
 NSTEP =  1440 CNT0   000000000  30.050  30.050  30.175   0.996               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         2860/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     53    0       46      1044068     2860 ************     5.6     1.3     4.3     1.9    10.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3132131.0 3132131.0    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3129040.4 3129040.4     99.90      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  105    4367.8    4367.8     14.64      0.00
   9 SPCM     - SPECTRAL COMP.                      96     868.1     868.1      2.66      0.00
  10 SCAN2M - PHYSICS                               97   17058.9   17058.9     52.83      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  8     727.1     727.1      0.19      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  5      48.6      48.6      0.01      0.00
  17 GRIDFPOS IN CNT4                                8      34.3      34.3      0.01      0.00
  27 RESTART FILES - WRITING                         2    2908.5    2908.5      0.19      0.00
  28 RESTART FILES - READING                         1    2564.4    2564.4      0.08      0.00
  29 SU4FPOS IN CNT4                                 8       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 8    6336.3    6336.3      1.62      0.00
  31 POSDDH IN STEPO                                 1       1.3       1.3      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     97    6315.9    6315.9     19.56      0.00
  39 SU0YOMB                                         1     314.3     314.3      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                     97      29.4      29.4      0.09      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               96     386.1     386.1      1.18      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               96      57.1      57.1      0.18      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  5       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      212     408.8     408.8      2.77      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       200     244.0     244.0      1.56      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        200      88.0      88.0      0.56      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       212     144.3     144.3      0.98      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      52.2      52.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            212      47.1      47.1      0.32      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            200      25.0      25.0      0.16      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            212      96.4      96.4      0.65      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            200      22.8      22.8      0.15      0.00
 400 GSTATS                                      12215       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 11311       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3132.1 CPU TIME    3123.5 VECTOR TIME     3123.5


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      414.2
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/slots/6/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/6/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12282594/ICMUAh7zg+001356
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
08:02:14 (773559): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0_r41715791_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Jun 2024 12:16:48 1509254 22434163 oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0 1,295,100 42,469 0.0328
07 Jun 2024 22:37:27 1509254 22434163 oifs_43r3_bl_a0lb_2016092300_20_1017_12282594_0 864,000 27,340 0.0316


©2024 cpdn.org