climateprediction.net home page
Task 22296178

Task 22296178

Name oifs_43r3_ps_0792_2015050100_123_984_12201436_1
Workunit 12201436
Created 7 Jan 2023, 8:18:20 UTC
Sent 7 Jan 2023, 8:31:57 UTC
Report deadline 6 Feb 2023, 8:31:57 UTC
Received 8 Jan 2023, 16:31:36 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1525488
Run time 12 hours 37 min 42 sec
CPU time 12 hours 32 min 10 sec
Validate state Invalid
Credit 7,257.60
Device peak FLOPS 4.53 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.59 MB
Peak swap size 4,978.02 MB
Peak disk usage 2,999.97 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
 34.724
  10:37:13 STEP 1246 H=1246:00 +CPU= 22.207
  10:37:35 STEP 1247 H=1247:00 +CPU= 22.159
  10:38:17 STEP 1248 H=1248:00 +CPU= 40.384
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_51.zip
Uploading the intermediate file: upload_file_51.zip
  10:38:39 STEP 1249 H=1249:00 +CPU= 21.531
  10:39:01 STEP 1250 H=1250:00 +CPU= 22.218
  10:39:36 STEP 1251 H=1251:00 +CPU= 34.708
  10:39:58 STEP 1252 H=1252:00 +CPU= 22.224
  10:40:20 STEP 1253 H=1253:00 +CPU= 21.572
  10:40:55 STEP 1254 H=1254:00 +CPU= 35.508
  10:41:17 STEP 1255 H=1255:00 +CPU= 21.586
  10:41:39 STEP 1256 H=1256:00 +CPU= 22.245
  10:42:14 STEP 1257 H=1257:00 +CPU= 34.664
  10:42:36 STEP 1258 H=1258:00 +CPU= 22.255
  10:42:58 STEP 1259 H=1259:00 +CPU= 21.622
  10:43:36 STEP 1260 H=1260:00 +CPU= 38.304
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001260
  10:43:58 STEP 1261 H=1261:00 +CPU= 21.614
  10:44:20 STEP 1262 H=1262:00 +CPU= 22.278
  10:44:55 STEP 1263 H=1263:00 +CPU= 34.744
  10:45:17 STEP 1264 H=1264:00 +CPU= 22.251
  10:45:39 STEP 1265 H=1265:00 +CPU= 21.575
  10:46:14 STEP 1266 H=1266:00 +CPU= 35.443
  10:46:36 STEP 1267 H=1267:00 +CPU= 21.541
  10:46:58 STEP 1268 H=1268:00 +CPU= 22.194
  10:47:32 STEP 1269 H=1269:00 +CPU= 34.655
  10:47:55 STEP 1270 H=1270:00 +CPU= 22.181
  10:48:17 STEP 1271 H=1271:00 +CPU= 22.126
  10:48:58 STEP 1272 H=1272:00 +CPU= 40.343
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_52.zip
Uploading the intermediate file: upload_file_52.zip
  10:49:20 STEP 1273 H=1273:00 +CPU= 21.558
  10:49:42 STEP 1274 H=1274:00 +CPU= 22.227
  10:50:17 STEP 1275 H=1275:00 +CPU= 34.724
  10:50:39 STEP 1276 H=1276:00 +CPU= 22.219
  10:51:01 STEP 1277 H=1277:00 +CPU= 21.538
  10:51:36 STEP 1278 H=1278:00 +CPU= 35.438
  10:51:58 STEP 1279 H=1279:00 +CPU= 21.566
  10:52:20 STEP 1280 H=1280:00 +CPU= 22.235
  10:52:55 STEP 1281 H=1281:00 +CPU= 34.692
  10:53:17 STEP 1282 H=1282:00 +CPU= 22.234
  10:53:39 STEP 1283 H=1283:00 +CPU= 21.569
  10:54:17 STEP 1284 H=1284:00 +CPU= 38.255
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001284
  10:54:39 STEP 1285 H=1285:00 +CPU= 21.577
  10:55:01 STEP 1286 H=1286:00 +CPU= 22.236
  10:55:36 STEP 1287 H=1287:00 +CPU= 34.693
  10:55:58 STEP 1288 H=1288:00 +CPU= 22.239
  10:56:19 STEP 1289 H=1289:00 +CPU= 21.566
  10:56:55 STEP 1290 H=1290:00 +CPU= 35.466
  10:57:16 STEP 1291 H=1291:00 +CPU= 21.581
  10:57:39 STEP 1292 H=1292:00 +CPU= 22.242
  10:58:13 STEP 1293 H=1293:00 +CPU= 34.731
  10:58:36 STEP 1294 H=1294:00 +CPU= 22.237
  10:58:58 STEP 1295 H=1295:00 +CPU= 22.181
  10:59:39 STEP 1296 H=1296:00 +CPU= 40.390
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_53.zip
Uploading the intermediate file: upload_file_53.zip
  11:00:01 STEP 1297 H=1297:00 +CPU= 21.543
  11:00:23 STEP 1298 H=1298:00 +CPU= 22.194
  11:00:58 STEP 1299 H=1299:00 +CPU= 34.693
  11:01:20 STEP 1300 H=1300:00 +CPU= 22.212
  11:01:42 STEP 1301 H=1301:00 +CPU= 21.556
  11:02:17 STEP 1302 H=1302:00 +CPU= 35.454
  11:02:39 STEP 1303 H=1303:00 +CPU= 21.577
  11:03:01 STEP 1304 H=1304:00 +CPU= 22.248
  11:03:36 STEP 1305 H=1305:00 +CPU= 34.709
  11:03:58 STEP 1306 H=1306:00 +CPU= 22.275
  11:04:20 STEP 1307 H=1307:00 +CPU= 21.618
  11:04:58 STEP 1308 H=1308:00 +CPU= 38.341
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001308
  11:05:20 STEP 1309 H=1309:00 +CPU= 21.612
  11:05:42 STEP 1310 H=1310:00 +CPU= 22.282
  11:06:17 STEP 1311 H=1311:00 +CPU= 34.725
  11:06:39 STEP 1312 H=1312:00 +CPU= 22.248
  11:07:01 STEP 1313 H=1313:00 +CPU= 21.579
  11:07:36 STEP 1314 H=1314:00 +CPU= 35.486
  11:07:58 STEP 1315 H=1315:00 +CPU= 21.594
  11:08:20 STEP 1316 H=1316:00 +CPU= 22.251
  11:08:55 STEP 1317 H=1317:00 +CPU= 34.731
  11:09:17 STEP 1318 H=1318:00 +CPU= 22.232
  11:09:40 STEP 1319 H=1319:00 +CPU= 22.181
  11:10:21 STEP 1320 H=1320:00 +CPU= 40.339
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_54.zip
Uploading the intermediate file: upload_file_54.zip
  11:10:42 STEP 1321 H=1321:00 +CPU= 21.527
  11:11:05 STEP 1322 H=1322:00 +CPU= 22.194
  11:11:39 STEP 1323 H=1323:00 +CPU= 34.677
  11:12:02 STEP 1324 H=1324:00 +CPU= 22.199
  11:12:23 STEP 1325 H=1325:00 +CPU= 21.552
  11:12:59 STEP 1326 H=1326:00 +CPU= 35.446
  11:13:20 STEP 1327 H=1327:00 +CPU= 21.581
  11:13:43 STEP 1328 H=1328:00 +CPU= 22.271
  11:14:17 STEP 1329 H=1329:00 +CPU= 34.754
  11:14:40 STEP 1330 H=1330:00 +CPU= 22.269
  11:15:01 STEP 1331 H=1331:00 +CPU= 21.608
  11:15:40 STEP 1332 H=1332:00 +CPU= 38.316
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001332
  11:16:01 STEP 1333 H=1333:00 +CPU= 21.627
  11:16:24 STEP 1334 H=1334:00 +CPU= 22.290
  11:16:58 STEP 1335 H=1335:00 +CPU= 34.766
  11:17:21 STEP 1336 H=1336:00 +CPU= 22.271
  11:17:42 STEP 1337 H=1337:00 +CPU= 21.601
  11:18:18 STEP 1338 H=1338:00 +CPU= 35.482
  11:18:39 STEP 1339 H=1339:00 +CPU= 21.550
  11:19:02 STEP 1340 H=1340:00 +CPU= 22.207
  11:19:36 STEP 1341 H=1341:00 +CPU= 34.676
  11:19:59 STEP 1342 H=1342:00 +CPU= 22.192
  11:20:21 STEP 1343 H=1343:00 +CPU= 22.146
  11:21:02 STEP 1344 H=1344:00 +CPU= 40.371
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_55.zip
Uploading the intermediate file: upload_file_55.zip
  11:21:24 STEP 1345 H=1345:00 +CPU= 21.560
  11:21:46 STEP 1346 H=1346:00 +CPU= 22.219
  11:22:21 STEP 1347 H=1347:00 +CPU= 34.699
  11:22:43 STEP 1348 H=1348:00 +CPU= 22.228
  11:23:05 STEP 1349 H=1349:00 +CPU= 21.513
  11:23:40 STEP 1350 H=1350:00 +CPU= 35.422
  11:24:02 STEP 1351 H=1351:00 +CPU= 21.603
  11:24:24 STEP 1352 H=1352:00 +CPU= 22.253
  11:24:59 STEP 1353 H=1353:00 +CPU= 34.720
  11:25:21 STEP 1354 H=1354:00 +CPU= 22.298
  11:25:43 STEP 1355 H=1355:00 +CPU= 21.629
  11:26:21 STEP 1356 H=1356:00 +CPU= 38.306
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001356
  11:26:43 STEP 1357 H=1357:00 +CPU= 21.624
  11:27:05 STEP 1358 H=1358:00 +CPU= 22.298
  11:27:40 STEP 1359 H=1359:00 +CPU= 34.695
  11:28:02 STEP 1360 H=1360:00 +CPU= 22.242
  11:28:24 STEP 1361 H=1361:00 +CPU= 21.567
  11:28:59 STEP 1362 H=1362:00 +CPU= 35.419
  11:29:21 STEP 1363 H=1363:00 +CPU= 21.567
  11:29:43 STEP 1364 H=1364:00 +CPU= 22.238
  11:30:18 STEP 1365 H=1365:00 +CPU= 34.680
  11:30:40 STEP 1366 H=1366:00 +CPU= 22.237
  11:31:02 STEP 1367 H=1367:00 +CPU= 22.198
  11:31:44 STEP 1368 H=1368:00 +CPU= 40.332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_56.zip
Uploading the intermediate file: upload_file_56.zip
  11:32:06 STEP 1369 H=1369:00 +CPU= 21.556
  11:32:28 STEP 1370 H=1370:00 +CPU= 22.243
  11:33:03 STEP 1371 H=1371:00 +CPU= 34.707
  11:33:25 STEP 1372 H=1372:00 +CPU= 22.254
  11:33:46 STEP 1373 H=1373:00 +CPU= 21.594
  11:34:22 STEP 1374 H=1374:00 +CPU= 35.483
  11:34:44 STEP 1375 H=1375:00 +CPU= 21.615
  11:35:06 STEP 1376 H=1376:00 +CPU= 22.290
  11:35:41 STEP 1377 H=1377:00 +CPU= 34.728
  11:36:03 STEP 1378 H=1378:00 +CPU= 22.263
  11:36:25 STEP 1379 H=1379:00 +CPU= 21.618
  11:37:03 STEP 1380 H=1380:00 +CPU= 38.323
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001380
  11:37:25 STEP 1381 H=1381:00 +CPU= 21.642
  11:37:47 STEP 1382 H=1382:00 +CPU= 22.309
  11:38:22 STEP 1383 H=1383:00 +CPU= 34.751
  11:38:44 STEP 1384 H=1384:00 +CPU= 22.279
  11:39:06 STEP 1385 H=1385:00 +CPU= 21.624
  11:39:41 STEP 1386 H=1386:00 +CPU= 35.438
  11:40:03 STEP 1387 H=1387:00 +CPU= 21.550
  11:40:25 STEP 1388 H=1388:00 +CPU= 22.244
  11:41:00 STEP 1389 H=1389:00 +CPU= 34.742
  11:41:22 STEP 1390 H=1390:00 +CPU= 22.245
  11:41:45 STEP 1391 H=1391:00 +CPU= 22.204
  11:42:26 STEP 1392 H=1392:00 +CPU= 40.344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_57.zip
Uploading the intermediate file: upload_file_57.zip
  11:42:48 STEP 1393 H=1393:00 +CPU= 21.531
  11:43:10 STEP 1394 H=1394:00 +CPU= 22.199
  11:43:45 STEP 1395 H=1395:00 +CPU= 34.651
  11:44:07 STEP 1396 H=1396:00 +CPU= 22.217
  11:44:28 STEP 1397 H=1397:00 +CPU= 21.558
  11:45:04 STEP 1398 H=1398:00 +CPU= 35.421
  11:45:25 STEP 1399 H=1399:00 +CPU= 21.580
  11:45:48 STEP 1400 H=1400:00 +CPU= 22.254
  11:46:22 STEP 1401 H=1401:00 +CPU= 34.690
  11:46:45 STEP 1402 H=1402:00 +CPU= 22.264
  11:47:06 STEP 1403 H=1403:00 +CPU= 21.599
  11:47:45 STEP 1404 H=1404:00 +CPU= 38.292
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001404
  11:48:06 STEP 1405 H=1405:00 +CPU= 21.658
  11:48:29 STEP 1406 H=1406:00 +CPU= 22.335
  11:49:03 STEP 1407 H=1407:00 +CPU= 34.774
  11:49:26 STEP 1408 H=1408:00 +CPU= 22.290
  11:49:47 STEP 1409 H=1409:00 +CPU= 21.604
  11:50:23 STEP 1410 H=1410:00 +CPU= 35.495
  11:50:45 STEP 1411 H=1411:00 +CPU= 21.595
  11:51:07 STEP 1412 H=1412:00 +CPU= 22.259
  11:51:42 STEP 1413 H=1413:00 +CPU= 34.746
  11:52:04 STEP 1414 H=1414:00 +CPU= 22.246
  11:52:26 STEP 1415 H=1415:00 +CPU= 22.200
  11:53:07 STEP 1416 H=1416:00 +CPU= 40.324
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_58.zip
Uploading the intermediate file: upload_file_58.zip
  11:53:29 STEP 1417 H=1417:00 +CPU= 21.527
  11:53:52 STEP 1418 H=1418:00 +CPU= 22.209
  11:54:26 STEP 1419 H=1419:00 +CPU= 34.642
  11:54:48 STEP 1420 H=1420:00 +CPU= 22.214
  11:55:10 STEP 1421 H=1421:00 +CPU= 21.584
  11:55:46 STEP 1422 H=1422:00 +CPU= 35.463
  11:56:07 STEP 1423 H=1423:00 +CPU= 21.574
  11:56:29 STEP 1424 H=1424:00 +CPU= 22.275
  11:57:04 STEP 1425 H=1425:00 +CPU= 34.603
  11:57:26 STEP 1426 H=1426:00 +CPU= 22.282
  11:57:48 STEP 1427 H=1427:00 +CPU= 21.616
  11:58:26 STEP 1428 H=1428:00 +CPU= 38.294
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001428
  11:58:48 STEP 1429 H=1429:00 +CPU= 21.627
  11:59:10 STEP 1430 H=1430:00 +CPU= 22.263
  11:59:45 STEP 1431 H=1431:00 +CPU= 34.690
  12:00:07 STEP 1432 H=1432:00 +CPU= 22.255
  12:00:29 STEP 1433 H=1433:00 +CPU= 21.590
  12:01:04 STEP 1434 H=1434:00 +CPU= 35.468
  12:01:26 STEP 1435 H=1435:00 +CPU= 21.551
  12:01:48 STEP 1436 H=1436:00 +CPU= 22.191
  12:02:23 STEP 1437 H=1437:00 +CPU= 34.639
  12:02:45 STEP 1438 H=1438:00 +CPU= 22.173
  12:03:07 STEP 1439 H=1439:00 +CPU= 22.122
  12:03:49 STEP 1440 H=1440:00 +CPU= 40.301
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_59.zip
Uploading the intermediate file: upload_file_59.zip
  12:04:10 STEP 1441 H=1441:00 +CPU= 21.517
Uploading trickle at timestep: 5184000
  12:04:33 STEP 1442 H=1442:00 +CPU= 22.190
  12:05:07 STEP 1443 H=1443:00 +CPU= 34.646
  12:05:30 STEP 1444 H=1444:00 +CPU= 22.214
  12:05:51 STEP 1445 H=1445:00 +CPU= 21.556
  12:06:27 STEP 1446 H=1446:00 +CPU= 35.424
  12:06:48 STEP 1447 H=1447:00 +CPU= 21.576
  12:07:10 STEP 1448 H=1448:00 +CPU= 22.249
  12:07:45 STEP 1449 H=1449:00 +CPU= 34.681
  12:08:07 STEP 1450 H=1450:00 +CPU= 22.258
  12:08:29 STEP 1451 H=1451:00 +CPU= 21.591
  12:09:07 STEP 1452 H=1452:00 +CPU= 38.268
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001452
  12:09:29 STEP 1453 H=1453:00 +CPU= 21.598
  12:09:51 STEP 1454 H=1454:00 +CPU= 22.260
  12:10:26 STEP 1455 H=1455:00 +CPU= 34.700
  12:10:48 STEP 1456 H=1456:00 +CPU= 22.237
  12:11:10 STEP 1457 H=1457:00 +CPU= 21.554
  12:11:45 STEP 1458 H=1458:00 +CPU= 35.410
  12:12:07 STEP 1459 H=1459:00 +CPU= 21.539
  12:12:29 STEP 1460 H=1460:00 +CPU= 22.202
  12:13:04 STEP 1461 H=1461:00 +CPU= 34.662
  12:13:26 STEP 1462 H=1462:00 +CPU= 22.183
  12:13:48 STEP 1463 H=1463:00 +CPU= 22.133
  12:14:30 STEP 1464 H=1464:00 +CPU= 40.349
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_60.zip
Uploading the intermediate file: upload_file_60.zip
  12:14:51 STEP 1465 H=1465:00 +CPU= 21.513
  12:15:14 STEP 1466 H=1466:00 +CPU= 22.183
  12:15:48 STEP 1467 H=1467:00 +CPU= 34.635
  12:16:11 STEP 1468 H=1468:00 +CPU= 22.222
  12:16:32 STEP 1469 H=1469:00 +CPU= 21.572
  12:17:08 STEP 1470 H=1470:00 +CPU= 35.473
  12:17:29 STEP 1471 H=1471:00 +CPU= 21.590
  12:17:52 STEP 1472 H=1472:00 +CPU= 22.257
  12:18:26 STEP 1473 H=1473:00 +CPU= 34.716
  12:18:49 STEP 1474 H=1474:00 +CPU= 22.258
  12:19:10 STEP 1475 H=1475:00 +CPU= 21.588
  12:19:49 STEP 1476 H=1476:00 +CPU= 38.251
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001476
  12:20:10 STEP 1477 H=1477:00 +CPU= 21.574
  12:20:33 STEP 1478 H=1478:00 +CPU= 22.259
  12:21:07 STEP 1479 H=1479:00 +CPU= 34.689
  12:21:30 STEP 1480 H=1480:00 +CPU= 22.239
  12:21:51 STEP 1481 H=1481:00 +CPU= 21.576
  12:22:27 STEP 1482 H=1482:00 +CPU= 35.478
  12:22:48 STEP 1483 H=1483:00 +CPU= 21.555
  12:23:10 STEP 1484 H=1484:00 +CPU= 22.211
  12:23:45 STEP 1485 H=1485:00 +CPU= 34.713
  12:24:07 STEP 1486 H=1486:00 +CPU= 22.221
  12:24:30 STEP 1487 H=1487:00 +CPU= 22.168
  12:25:11 STEP 1488 H=1488:00 +CPU= 40.331
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_61.zip
Uploading the intermediate file: upload_file_61.zip
  12:25:33 STEP 1489 H=1489:00 +CPU= 21.537
  12:25:55 STEP 1490 H=1490:00 +CPU= 22.224
  12:26:30 STEP 1491 H=1491:00 +CPU= 34.725
  12:26:52 STEP 1492 H=1492:00 +CPU= 22.234
  12:27:14 STEP 1493 H=1493:00 +CPU= 21.567
  12:27:49 STEP 1494 H=1494:00 +CPU= 35.504
  12:28:11 STEP 1495 H=1495:00 +CPU= 21.586
  12:28:33 STEP 1496 H=1496:00 +CPU= 22.264
  12:29:08 STEP 1497 H=1497:00 +CPU= 34.747
  12:29:30 STEP 1498 H=1498:00 +CPU= 22.256
  12:29:52 STEP 1499 H=1499:00 +CPU= 21.588
  12:30:30 STEP 1500 H=1500:00 +CPU= 38.279
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001500
  12:30:52 STEP 1501 H=1501:00 +CPU= 21.592
  12:31:14 STEP 1502 H=1502:00 +CPU= 22.260
  12:31:49 STEP 1503 H=1503:00 +CPU= 34.720
  12:32:11 STEP 1504 H=1504:00 +CPU= 22.232
  12:32:33 STEP 1505 H=1505:00 +CPU= 21.561
  12:33:08 STEP 1506 H=1506:00 +CPU= 35.499
  12:33:30 STEP 1507 H=1507:00 +CPU= 21.585
  12:33:52 STEP 1508 H=1508:00 +CPU= 22.236
  12:34:27 STEP 1509 H=1509:00 +CPU= 34.707
  12:34:49 STEP 1510 H=1510:00 +CPU= 22.206
  12:35:11 STEP 1511 H=1511:00 +CPU= 22.159
  12:35:53 STEP 1512 H=1512:00 +CPU= 40.346
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_62.zip
Uploading the intermediate file: upload_file_62.zip
  12:36:15 STEP 1513 H=1513:00 +CPU= 21.525
  12:36:37 STEP 1514 H=1514:00 +CPU= 22.218
  12:37:12 STEP 1515 H=1515:00 +CPU= 34.711
  12:37:34 STEP 1516 H=1516:00 +CPU= 22.220
  12:37:56 STEP 1517 H=1517:00 +CPU= 21.547
  12:38:31 STEP 1518 H=1518:00 +CPU= 35.402
  12:38:53 STEP 1519 H=1519:00 +CPU= 21.489
  12:39:15 STEP 1520 H=1520:00 +CPU= 22.219
  12:39:49 STEP 1521 H=1521:00 +CPU= 34.681
  12:40:12 STEP 1522 H=1522:00 +CPU= 22.225
  12:40:33 STEP 1523 H=1523:00 +CPU= 21.566
  12:41:12 STEP 1524 H=1524:00 +CPU= 38.261
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001524
  12:41:33 STEP 1525 H=1525:00 +CPU= 21.564
  12:41:56 STEP 1526 H=1526:00 +CPU= 22.225
  12:42:30 STEP 1527 H=1527:00 +CPU= 34.651
  12:42:52 STEP 1528 H=1528:00 +CPU= 22.191
  12:43:14 STEP 1529 H=1529:00 +CPU= 21.518
  12:43:49 STEP 1530 H=1530:00 +CPU= 35.397
  12:44:11 STEP 1531 H=1531:00 +CPU= 21.503
  12:44:33 STEP 1532 H=1532:00 +CPU= 22.158
  12:45:08 STEP 1533 H=1533:00 +CPU= 34.654
  12:45:30 STEP 1534 H=1534:00 +CPU= 22.161
  12:45:52 STEP 1535 H=1535:00 +CPU= 22.099
  12:46:33 STEP 1536 H=1536:00 +CPU= 40.264
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_63.zip
Uploading the intermediate file: upload_file_63.zip
  12:46:55 STEP 1537 H=1537:00 +CPU= 21.498
  12:47:17 STEP 1538 H=1538:00 +CPU= 22.155
  12:47:52 STEP 1539 H=1539:00 +CPU= 34.581
  12:48:14 STEP 1540 H=1540:00 +CPU= 22.176
  12:48:36 STEP 1541 H=1541:00 +CPU= 21.525
  12:49:11 STEP 1542 H=1542:00 +CPU= 35.406
  12:49:33 STEP 1543 H=1543:00 +CPU= 21.543
  12:49:55 STEP 1544 H=1544:00 +CPU= 22.216
  12:50:30 STEP 1545 H=1545:00 +CPU= 34.674
  12:50:52 STEP 1546 H=1546:00 +CPU= 22.226
  12:51:14 STEP 1547 H=1547:00 +CPU= 21.568
  12:51:52 STEP 1548 H=1548:00 +CPU= 38.270
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001548
  12:52:13 STEP 1549 H=1549:00 +CPU= 21.582
  12:52:36 STEP 1550 H=1550:00 +CPU= 22.254
  12:53:10 STEP 1551 H=1551:00 +CPU= 34.714
  12:53:33 STEP 1552 H=1552:00 +CPU= 22.223
  12:53:54 STEP 1553 H=1553:00 +CPU= 21.564
  12:54:30 STEP 1554 H=1554:00 +CPU= 35.497
  12:54:51 STEP 1555 H=1555:00 +CPU= 21.549
  12:55:14 STEP 1556 H=1556:00 +CPU= 22.206
  12:55:48 STEP 1557 H=1557:00 +CPU= 34.711
  12:56:11 STEP 1558 H=1558:00 +CPU= 22.208
  12:56:33 STEP 1559 H=1559:00 +CPU= 22.132
  12:57:14 STEP 1560 H=1560:00 +CPU= 40.380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_64.zip
Uploading the intermediate file: upload_file_64.zip
  12:57:36 STEP 1561 H=1561:00 +CPU= 21.554
  12:57:58 STEP 1562 H=1562:00 +CPU= 22.223
  12:58:33 STEP 1563 H=1563:00 +CPU= 34.715
  12:58:55 STEP 1564 H=1564:00 +CPU= 22.231
  12:59:17 STEP 1565 H=1565:00 +CPU= 21.554
  12:59:52 STEP 1566 H=1566:00 +CPU= 35.427
  13:00:14 STEP 1567 H=1567:00 +CPU= 21.572
  13:00:36 STEP 1568 H=1568:00 +CPU= 22.239
  13:01:11 STEP 1569 H=1569:00 +CPU= 34.692
  13:01:33 STEP 1570 H=1570:00 +CPU= 22.240
  13:01:55 STEP 1571 H=1571:00 +CPU= 21.578
  13:02:33 STEP 1572 H=1572:00 +CPU= 38.272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001572
  13:02:55 STEP 1573 H=1573:00 +CPU= 21.600
  13:03:17 STEP 1574 H=1574:00 +CPU= 22.277
  13:03:52 STEP 1575 H=1575:00 +CPU= 34.722
  13:04:14 STEP 1576 H=1576:00 +CPU= 22.237
  13:04:36 STEP 1577 H=1577:00 +CPU= 21.565
  13:05:11 STEP 1578 H=1578:00 +CPU= 35.479
  13:05:33 STEP 1579 H=1579:00 +CPU= 21.567
  13:05:55 STEP 1580 H=1580:00 +CPU= 22.222
  13:06:30 STEP 1581 H=1581:00 +CPU= 34.706
  13:06:52 STEP 1582 H=1582:00 +CPU= 22.204
  13:07:14 STEP 1583 H=1583:00 +CPU= 22.152
  13:07:55 STEP 1584 H=1584:00 +CPU= 40.332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_65.zip
Uploading the intermediate file: upload_file_65.zip
  13:08:17 STEP 1585 H=1585:00 +CPU= 21.516
  13:08:39 STEP 1586 H=1586:00 +CPU= 22.180
  13:09:14 STEP 1587 H=1587:00 +CPU= 34.646
  13:09:36 STEP 1588 H=1588:00 +CPU= 22.211
  13:09:58 STEP 1589 H=1589:00 +CPU= 21.533
  13:10:33 STEP 1590 H=1590:00 +CPU= 35.450
  13:10:55 STEP 1591 H=1591:00 +CPU= 21.596
  13:11:17 STEP 1592 H=1592:00 +CPU= 22.262
  13:11:52 STEP 1593 H=1593:00 +CPU= 34.743
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0792
(argv4) batchid: 984
(argv5) wuid: 12201436
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_da3ef340c5b8de830f199ec631ebf054 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0792_2015050100_123_984_12201436.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0792_2015050100_123_984_12201436.zip
ic_ancil_file: ic_ancil_12201436
ifsdata_file: ifsdata_12201436
climate_data_file: clim_data_12201436
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3373fded03144616d4e82f2cdf85f730 to: /var/lib/boinc-client/slots/3/ic_ancil_12201436.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12201436.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_82a2bd4f18467e09cc76c14d47fd238b to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12201436.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12201436.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12201436.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12201436.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12201436.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12201436.xml
-- Model is restarting --
Adjusting last_iter, 1593, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12201436.xml
last_cpu_time: 42454
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227
The child process has been launched with process id: 62501
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:62501:62501] [20230108:021600:1673172960:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:17:10 STEP 1584 H=1584:00 +CPU= 40.100
  02:17:43 STEP 1585 H=1585:00 +CPU= 21.630
  02:18:05 STEP 1586 H=1586:00 +CPU= 21.498
  02:18:40 STEP 1587 H=1587:00 +CPU= 33.763
  02:19:02 STEP 1588 H=1588:00 +CPU= 22.209
  02:19:24 STEP 1589 H=1589:00 +CPU= 21.558
  02:19:59 STEP 1590 H=1590:00 +CPU= 35.410
  02:20:21 STEP 1591 H=1591:00 +CPU= 21.558
  02:20:43 STEP 1592 H=1592:00 +CPU= 22.250
  02:21:18 STEP 1593 H=1593:00 +CPU= 34.645
  02:21:40 STEP 1594 H=1594:00 +CPU= 22.231
  02:22:02 STEP 1595 H=1595:00 +CPU= 21.568
  02:22:41 STEP 1596 H=1596:00 +CPU= 38.206
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001596
  02:23:02 STEP 1597 H=1597:00 +CPU= 21.579
  02:23:25 STEP 1598 H=1598:00 +CPU= 22.239
  02:23:59 STEP 1599 H=1599:00 +CPU= 34.638
  02:24:22 STEP 1600 H=1600:00 +CPU= 22.218
  02:24:43 STEP 1601 H=1601:00 +CPU= 21.549
  02:25:19 STEP 1602 H=1602:00 +CPU= 35.376
  02:25:40 STEP 1603 H=1603:00 +CPU= 21.529
  02:26:02 STEP 1604 H=1604:00 +CPU= 22.203
  02:26:37 STEP 1605 H=1605:00 +CPU= 34.649
  02:26:59 STEP 1606 H=1606:00 +CPU= 22.187
  02:27:22 STEP 1607 H=1607:00 +CPU= 22.117
  02:28:03 STEP 1608 H=1608:00 +CPU= 40.130
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_66.zip
Uploading the intermediate file: upload_file_66.zip
  02:28:25 STEP 1609 H=1609:00 +CPU= 21.503
  02:28:47 STEP 1610 H=1610:00 +CPU= 22.170
  02:29:22 STEP 1611 H=1611:00 +CPU= 34.604
  02:29:44 STEP 1612 H=1612:00 +CPU= 22.182
  02:30:05 STEP 1613 H=1613:00 +CPU= 21.528
  02:30:41 STEP 1614 H=1614:00 +CPU= 35.374
  02:31:02 STEP 1615 H=1615:00 +CPU= 21.552
  02:31:25 STEP 1616 H=1616:00 +CPU= 22.221
  02:31:59 STEP 1617 H=1617:00 +CPU= 34.692
  02:32:22 STEP 1618 H=1618:00 +CPU= 22.273
  02:32:43 STEP 1619 H=1619:00 +CPU= 21.607
  02:33:22 STEP 1620 H=1620:00 +CPU= 38.246
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001620
  02:33:43 STEP 1621 H=1621:00 +CPU= 21.540
  02:34:05 STEP 1622 H=1622:00 +CPU= 22.209
  02:34:40 STEP 1623 H=1623:00 +CPU= 34.616
  02:35:02 STEP 1624 H=1624:00 +CPU= 22.182
  02:35:24 STEP 1625 H=1625:00 +CPU= 21.516
  02:35:59 STEP 1626 H=1626:00 +CPU= 35.333
  02:36:21 STEP 1627 H=1627:00 +CPU= 21.601
  02:36:43 STEP 1628 H=1628:00 +CPU= 22.156
  02:37:17 STEP 1629 H=1629:00 +CPU= 34.582
  02:37:40 STEP 1630 H=1630:00 +CPU= 22.142
  02:38:02 STEP 1631 H=1631:00 +CPU= 22.135
  02:38:43 STEP 1632 H=1632:00 +CPU= 40.288
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_67.zip
Uploading the intermediate file: upload_file_67.zip
  02:39:04 STEP 1633 H=1633:00 +CPU= 21.529
  02:39:27 STEP 1634 H=1634:00 +CPU= 22.179
  02:40:01 STEP 1635 H=1635:00 +CPU= 34.611
  02:40:23 STEP 1636 H=1636:00 +CPU= 22.200
  02:40:45 STEP 1637 H=1637:00 +CPU= 21.564
  02:41:21 STEP 1638 H=1638:00 +CPU= 35.442
  02:41:42 STEP 1639 H=1639:00 +CPU= 21.554
  02:42:04 STEP 1640 H=1640:00 +CPU= 22.216
  02:42:39 STEP 1641 H=1641:00 +CPU= 34.624
  02:43:01 STEP 1642 H=1642:00 +CPU= 22.231
  02:43:23 STEP 1643 H=1643:00 +CPU= 21.598
  02:44:01 STEP 1644 H=1644:00 +CPU= 38.256
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001644
  02:44:23 STEP 1645 H=1645:00 +CPU= 21.614
  02:44:45 STEP 1646 H=1646:00 +CPU= 22.279
  02:45:20 STEP 1647 H=1647:00 +CPU= 34.684
  02:45:42 STEP 1648 H=1648:00 +CPU= 22.224
  02:46:04 STEP 1649 H=1649:00 +CPU= 21.547
  02:46:39 STEP 1650 H=1650:00 +CPU= 35.363
  02:47:01 STEP 1651 H=1651:00 +CPU= 21.527
  02:47:23 STEP 1652 H=1652:00 +CPU= 22.182
  02:47:57 STEP 1653 H=1653:00 +CPU= 34.586
  02:48:20 STEP 1654 H=1654:00 +CPU= 22.173
  02:48:42 STEP 1655 H=1655:00 +CPU= 22.113
  02:49:23 STEP 1656 H=1656:00 +CPU= 40.199
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_68.zip
Uploading the intermediate file: upload_file_68.zip
  02:49:44 STEP 1657 H=1657:00 +CPU= 21.490
  02:50:06 STEP 1658 H=1658:00 +CPU= 22.161
  02:50:41 STEP 1659 H=1659:00 +CPU= 34.564
  02:51:03 STEP 1660 H=1660:00 +CPU= 22.211
  02:51:25 STEP 1661 H=1661:00 +CPU= 21.553
  02:52:00 STEP 1662 H=1662:00 +CPU= 35.341
  02:52:22 STEP 1663 H=1663:00 +CPU= 21.541
  02:52:44 STEP 1664 H=1664:00 +CPU= 22.207
  02:53:19 STEP 1665 H=1665:00 +CPU= 34.591
  02:53:41 STEP 1666 H=1666:00 +CPU= 22.217
  02:54:02 STEP 1667 H=1667:00 +CPU= 21.589
  02:54:41 STEP 1668 H=1668:00 +CPU= 38.204
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001668
  02:55:02 STEP 1669 H=1669:00 +CPU= 21.617
  02:55:25 STEP 1670 H=1670:00 +CPU= 22.278
  02:55:59 STEP 1671 H=1671:00 +CPU= 34.653
  02:56:22 STEP 1672 H=1672:00 +CPU= 22.247
  02:56:43 STEP 1673 H=1673:00 +CPU= 21.445
  02:57:35 STEP 1674 H=1674:00 +CPU= 35.561
  02:57:56 STEP 1675 H=1675:00 +CPU= 20.870
  02:58:17 STEP 1676 H=1676:00 +CPU= 21.619
  07:13:57 STEP 1677 H=1677:00 +CPU= 39.315
  07:24:43 STEP 1678 H=1678:00 +CPU= 22.917
  07:25:13 STEP 1679 H=1679:00 +CPU= 21.923
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0792
(argv4) batchid: 984
(argv5) wuid: 12201436
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_da3ef340c5b8de830f199ec631ebf054 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0792_2015050100_123_984_12201436.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0792_2015050100_123_984_12201436.zip
ic_ancil_file: ic_ancil_12201436
ifsdata_file: ifsdata_12201436
climate_data_file: clim_data_12201436
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_3373fded03144616d4e82f2cdf85f730 to: /var/lib/boinc-client/slots/3/ic_ancil_12201436.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12201436.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_82a2bd4f18467e09cc76c14d47fd238b to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12201436.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12201436.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12201436.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12201436.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12201436.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12201436.xml
-- Model is restarting --
Adjusting last_iter, 1680, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12201436.xml
last_cpu_time: 45027
upload_file_number: 69
last_iter: 1679
last_upload: 5961600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227
The child process has been launched with process id: 63490
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:CB-BOINC-07:1:1:63490:63490] [20230108:072726:1673191646:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:28:34 STEP 1680 H=1680:00 +CPU= 38.924
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12201436/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0792_2015050100_123_984_12201436_1_r1800225227_69.zip
Uploading the intermediate file: upload_file_69.zip
  07:28:55 STEP 1681 H=1681:00 +CPU= 20.849
  07:29:16 STEP 1682 H=1682:00 +CPU= 21.666
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:12:03 1525488 22296178 oifs_43r3_ps_0792_2015050100_123_984_12201436_1 5,184,000 38,393 0.0074
09 May 2023 16:11:58 1525488 22296178 oifs_43r3_ps_0792_2015050100_123_984_12201436_1 4,320,000 31,998 0.0074
09 May 2023 16:11:53 1525488 22296178 oifs_43r3_ps_0792_2015050100_123_984_12201436_1 3,456,000 25,612 0.0074
09 May 2023 16:11:48 1525488 22296178 oifs_43r3_ps_0792_2015050100_123_984_12201436_1 2,592,000 19,214 0.0074
09 May 2023 16:11:43 1525488 22296178 oifs_43r3_ps_0792_2015050100_123_984_12201436_1 1,728,000 12,821 0.0074
09 May 2023 16:11:38 1525488 22296178 oifs_43r3_ps_0792_2015050100_123_984_12201436_1 864,000 6,432 0.0074


©2024 climateprediction.net