climateprediction.net home page
Task 22449189

Task 22449189

Name oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2
Workunit 12290011
Created 15 Jun 2024, 17:05:57 UTC
Sent 15 Jun 2024, 17:06:36 UTC
Report deadline 14 Aug 2024, 17:06:36 UTC
Received 17 Jun 2024, 5:39:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1536326
Run time 1 days 11 hours 28 min 24 sec
CPU time 1 days 11 hours 4 min 22 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 2.41 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.15 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,357.56 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
r3_bl_12290011/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2_r391903846_13.zip
Uploading the intermediate file: upload_file_13.zip
  12:14:28 STEP 1345 H= 336:15 +CPU= 75.954
  12:15:38 STEP 1346 H= 336:30 +CPU= 68.348
  12:16:44 STEP 1347 H= 336:45 +CPU= 66.074
  12:17:51 STEP 1348 H= 337:00 +CPU= 66.392
  12:18:56 STEP 1349 H= 337:15 +CPU= 64.972
  12:19:58 STEP 1350 H= 337:30 +CPU= 62.478
  12:21:01 STEP 1351 H= 337:45 +CPU= 62.715
  12:22:06 STEP 1352 H= 338:00 +CPU= 65.355
  12:23:11 STEP 1353 H= 338:15 +CPU= 64.769
  12:24:15 STEP 1354 H= 338:30 +CPU= 64.333
  12:25:19 STEP 1355 H= 338:45 +CPU= 63.696
  12:26:38 STEP 1356 H= 339:00 +CPU= 78.365
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001356
  12:27:40 STEP 1357 H= 339:15 +CPU= 62.521
  12:28:43 STEP 1358 H= 339:30 +CPU= 62.664
  12:29:47 STEP 1359 H= 339:45 +CPU= 63.734
  12:30:52 STEP 1360 H= 340:00 +CPU= 65.473
  12:31:58 STEP 1361 H= 340:15 +CPU= 65.380
  12:33:02 STEP 1362 H= 340:30 +CPU= 64.311
  12:34:08 STEP 1363 H= 340:45 +CPU= 65.672
  12:35:15 STEP 1364 H= 341:00 +CPU= 67.113
  12:36:21 STEP 1365 H= 341:15 +CPU= 65.435
  12:37:27 STEP 1366 H= 341:30 +CPU= 66.140
  12:38:32 STEP 1367 H= 341:45 +CPU= 65.496
  12:39:54 STEP 1368 H= 342:00 +CPU= 82.078
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001368
  12:40:58 STEP 1369 H= 342:15 +CPU= 63.392
  12:42:03 STEP 1370 H= 342:30 +CPU= 64.955
  12:43:07 STEP 1371 H= 342:45 +CPU= 64.601
  12:44:13 STEP 1372 H= 343:00 +CPU= 65.477
  12:45:15 STEP 1373 H= 343:15 +CPU= 62.411
  12:46:18 STEP 1374 H= 343:30 +CPU= 63.030
  12:47:21 STEP 1375 H= 343:45 +CPU= 62.430
  12:48:25 STEP 1376 H= 344:00 +CPU= 64.294
  12:49:29 STEP 1377 H= 344:15 +CPU= 63.491
  12:50:31 STEP 1378 H= 344:30 +CPU= 62.866
  12:51:34 STEP 1379 H= 344:45 +CPU= 62.957
  12:53:00 STEP 1380 H= 345:00 +CPU= 85.716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001380
  12:54:10 STEP 1381 H= 345:15 +CPU= 69.347
  12:55:19 STEP 1382 H= 345:30 +CPU= 69.262
  12:56:29 STEP 1383 H= 345:45 +CPU= 69.554
  12:57:38 STEP 1384 H= 346:00 +CPU= 68.857
  12:58:46 STEP 1385 H= 346:15 +CPU= 67.682
  12:59:53 STEP 1386 H= 346:30 +CPU= 66.345
  13:00:57 STEP 1387 H= 346:45 +CPU= 63.835
  13:02:03 STEP 1388 H= 347:00 +CPU= 66.195
  13:03:09 STEP 1389 H= 347:15 +CPU= 66.306
  13:04:14 STEP 1390 H= 347:30 +CPU= 64.662
  13:05:17 STEP 1391 H= 347:45 +CPU= 63.008
  13:06:45 STEP 1392 H= 348:00 +CPU= 87.761
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001392
  13:07:50 STEP 1393 H= 348:15 +CPU= 64.934
  13:08:56 STEP 1394 H= 348:30 +CPU= 66.364
  13:10:03 STEP 1395 H= 348:45 +CPU= 67.094
  13:11:08 STEP 1396 H= 349:00 +CPU= 64.952
  13:12:14 STEP 1397 H= 349:15 +CPU= 65.071
  13:13:19 STEP 1398 H= 349:30 +CPU= 65.090
  13:14:23 STEP 1399 H= 349:45 +CPU= 64.313
  13:15:32 STEP 1400 H= 350:00 +CPU= 68.529
  13:16:37 STEP 1401 H= 350:15 +CPU= 65.041
  13:17:45 STEP 1402 H= 350:30 +CPU= 67.882
  13:18:56 STEP 1403 H= 350:45 +CPU= 70.719
  13:20:22 STEP 1404 H= 351:00 +CPU= 86.056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001404
  13:21:33 STEP 1405 H= 351:15 +CPU= 71.129
  13:22:42 STEP 1406 H= 351:30 +CPU= 69.174
  13:23:51 STEP 1407 H= 351:45 +CPU= 68.980
  13:25:01 STEP 1408 H= 352:00 +CPU= 69.041
  13:26:13 STEP 1409 H= 352:15 +CPU= 72.054
  13:27:26 STEP 1410 H= 352:30 +CPU= 73.181
  13:28:39 STEP 1411 H= 352:45 +CPU= 73.099
  13:29:50 STEP 1412 H= 353:00 +CPU= 70.161
  13:30:57 STEP 1413 H= 353:15 +CPU= 66.959
  13:32:03 STEP 1414 H= 353:30 +CPU= 66.814
  13:33:09 STEP 1415 H= 353:45 +CPU= 65.538
  13:34:31 STEP 1416 H= 354:00 +CPU= 81.554
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001416
  13:35:37 STEP 1417 H= 354:15 +CPU= 66.090
  13:36:45 STEP 1418 H= 354:30 +CPU= 68.093
  13:37:50 STEP 1419 H= 354:45 +CPU= 65.150
  13:38:55 STEP 1420 H= 355:00 +CPU= 65.299
  13:40:00 STEP 1421 H= 355:15 +CPU= 65.071
  13:41:08 STEP 1422 H= 355:30 +CPU= 67.304
  13:42:16 STEP 1423 H= 355:45 +CPU= 67.817
  13:43:25 STEP 1424 H= 356:00 +CPU= 69.675
  13:44:37 STEP 1425 H= 356:15 +CPU= 72.065
  13:45:50 STEP 1426 H= 356:30 +CPU= 72.623
  13:47:02 STEP 1427 H= 356:45 +CPU= 71.547
  13:48:10 STEP 1428 H= 357:00 +CPU= 68.394
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001428
  13:49:18 STEP 1429 H= 357:15 +CPU= 68.096
  13:50:29 STEP 1430 H= 357:30 +CPU= 70.603
  13:51:42 STEP 1431 H= 357:45 +CPU= 72.704
  13:52:55 STEP 1432 H= 358:00 +CPU= 73.644
  13:54:11 STEP 1433 H= 358:15 +CPU= 75.161
  13:55:28 STEP 1434 H= 358:30 +CPU= 77.050
  13:56:43 STEP 1435 H= 358:45 +CPU= 74.394
  13:57:58 STEP 1436 H= 359:00 +CPU= 74.749
  13:59:11 STEP 1437 H= 359:15 +CPU= 72.596
  14:00:25 STEP 1438 H= 359:30 +CPU= 74.086
  14:01:38 STEP 1439 H= 359:45 +CPU= 73.198
  14:03:14 STEP 1440 H= 360:00 +CPU= 94.589
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2_r391903846_14.zip
Uploading the intermediate file: upload_file_14.zip
  14:04:39 STEP 1441 H= 360:15 +CPU= 81.541
  14:05:49 STEP 1442 H= 360:30 +CPU= 70.197
  14:06:56 STEP 1443 H= 360:45 +CPU= 67.132
  14:08:05 STEP 1444 H= 361:00 +CPU= 68.637
  14:09:34 STEP 1445 H= 361:15 +CPU= 88.764
  14:11:00 STEP 1446 H= 361:30 +CPU= 85.258
  14:12:13 STEP 1447 H= 361:45 +CPU= 72.402
  14:13:22 STEP 1448 H= 362:00 +CPU= 68.982
  14:14:31 STEP 1449 H= 362:15 +CPU= 69.474
  14:15:44 STEP 1450 H= 362:30 +CPU= 71.265
  14:16:53 STEP 1451 H= 362:45 +CPU= 68.848
  14:18:20 STEP 1452 H= 363:00 +CPU= 87.504
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001452
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001452
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001452
  14:19:30 STEP 1453 H= 363:15 +CPU= 68.860
  14:20:41 STEP 1454 H= 363:30 +CPU= 71.341
  14:21:51 STEP 1455 H= 363:45 +CPU= 70.090
  14:23:01 STEP 1456 H= 364:00 +CPU= 69.811
  14:24:11 STEP 1457 H= 364:15 +CPU= 69.556
  14:25:19 STEP 1458 H= 364:30 +CPU= 68.444
  14:26:30 STEP 1459 H= 364:45 +CPU= 70.427
  14:27:42 STEP 1460 H= 365:00 +CPU= 72.053
  14:29:06 STEP 1461 H= 365:15 +CPU= 84.451
  14:30:34 STEP 1462 H= 365:30 +CPU= 87.421
  14:31:44 STEP 1463 H= 365:45 +CPU= 70.144
  14:33:18 STEP 1464 H= 366:00 +CPU= 94.065
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001464
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001464
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001464
  14:34:33 STEP 1465 H= 366:15 +CPU= 74.208
  14:35:47 STEP 1466 H= 366:30 +CPU= 73.528
  14:36:55 STEP 1467 H= 366:45 +CPU= 68.722
  14:38:06 STEP 1468 H= 367:00 +CPU= 70.316
  14:39:17 STEP 1469 H= 367:15 +CPU= 71.039
  14:40:26 STEP 1470 H= 367:30 +CPU= 68.886
  14:41:37 STEP 1471 H= 367:45 +CPU= 71.384
  14:42:47 STEP 1472 H= 368:00 +CPU= 69.555
  14:43:55 STEP 1473 H= 368:15 +CPU= 67.958
  14:45:06 STEP 1474 H= 368:30 +CPU= 70.268
  14:46:10 STEP 1475 H= 368:45 +CPU= 64.855
  14:47:33 STEP 1476 H= 369:00 +CPU= 82.870
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001476
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001476
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001476
  14:48:38 STEP 1477 H= 369:15 +CPU= 64.879
  14:49:46 STEP 1478 H= 369:30 +CPU= 67.065
  14:50:52 STEP 1479 H= 369:45 +CPU= 66.227
  14:52:02 STEP 1480 H= 370:00 +CPU= 69.989
  14:53:10 STEP 1481 H= 370:15 +CPU= 67.622
  14:54:17 STEP 1482 H= 370:30 +CPU= 67.163
  14:55:24 STEP 1483 H= 370:45 +CPU= 66.423
  14:56:33 STEP 1484 H= 371:00 +CPU= 69.099
  14:57:43 STEP 1485 H= 371:15 +CPU= 69.876
  14:58:52 STEP 1486 H= 371:30 +CPU= 67.558
  15:00:01 STEP 1487 H= 371:45 +CPU= 69.529
  15:01:40 STEP 1488 H= 372:00 +CPU= 97.976
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001488
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001488
  15:02:52 STEP 1489 H= 372:15 +CPU= 72.069
  15:04:06 STEP 1490 H= 372:30 +CPU= 74.051
  15:05:18 STEP 1491 H= 372:45 +CPU= 71.219
  15:06:34 STEP 1492 H= 373:00 +CPU= 76.518
  15:07:48 STEP 1493 H= 373:15 +CPU= 73.666
  15:08:58 STEP 1494 H= 373:30 +CPU= 69.776
  15:10:05 STEP 1495 H= 373:45 +CPU= 67.154
  15:11:15 STEP 1496 H= 374:00 +CPU= 69.721
  15:12:26 STEP 1497 H= 374:15 +CPU= 71.384
  15:13:34 STEP 1498 H= 374:30 +CPU= 67.065
  15:14:46 STEP 1499 H= 374:45 +CPU= 71.701
  15:16:14 STEP 1500 H= 375:00 +CPU= 87.814
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001500
  15:17:23 STEP 1501 H= 375:15 +CPU= 69.482
  15:18:36 STEP 1502 H= 375:30 +CPU= 72.071
  15:19:47 STEP 1503 H= 375:45 +CPU= 71.188
  15:21:00 STEP 1504 H= 376:00 +CPU= 72.982
  15:22:06 STEP 1505 H= 376:15 +CPU= 66.117
  15:23:11 STEP 1506 H= 376:30 +CPU= 65.063
  15:24:23 STEP 1507 H= 376:45 +CPU= 71.833
  15:25:39 STEP 1508 H= 377:00 +CPU= 75.790
  15:26:50 STEP 1509 H= 377:15 +CPU= 71.592
  15:28:06 STEP 1510 H= 377:30 +CPU= 75.946
  15:29:21 STEP 1511 H= 377:45 +CPU= 74.517
  15:30:52 STEP 1512 H= 378:00 +CPU= 90.627
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001512
  15:32:05 STEP 1513 H= 378:15 +CPU= 73.303
  15:33:17 STEP 1514 H= 378:30 +CPU= 71.826
  15:34:28 STEP 1515 H= 378:45 +CPU= 71.586
  15:35:43 STEP 1516 H= 379:00 +CPU= 74.184
  15:36:54 STEP 1517 H= 379:15 +CPU= 71.838
  15:38:09 STEP 1518 H= 379:30 +CPU= 74.725
  15:39:23 STEP 1519 H= 379:45 +CPU= 74.147
  15:40:38 STEP 1520 H= 380:00 +CPU= 75.113
  15:41:52 STEP 1521 H= 380:15 +CPU= 73.186
  15:43:06 STEP 1522 H= 380:30 +CPU= 74.147
  15:44:21 STEP 1523 H= 380:45 +CPU= 74.599
  15:45:54 STEP 1524 H= 381:00 +CPU= 93.243
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001524
  15:47:08 STEP 1525 H= 381:15 +CPU= 73.618
  15:48:23 STEP 1526 H= 381:30 +CPU= 74.622
  15:49:28 STEP 1527 H= 381:45 +CPU= 65.849
  15:50:35 STEP 1528 H= 382:00 +CPU= 66.147
  15:51:39 STEP 1529 H= 382:15 +CPU= 64.652
  15:52:46 STEP 1530 H= 382:30 +CPU= 66.662
  15:53:53 STEP 1531 H= 382:45 +CPU= 66.747
  15:55:04 STEP 1532 H= 383:00 +CPU= 70.945
  15:56:18 STEP 1533 H= 383:15 +CPU= 73.471
  15:57:30 STEP 1534 H= 383:30 +CPU= 72.222
  15:58:41 STEP 1535 H= 383:45 +CPU= 69.796
  16:00:19 STEP 1536 H= 384:00 +CPU= 97.026
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2_r391903846_15.zip
Uploading the intermediate file: upload_file_15.zip
  16:01:40 STEP 1537 H= 384:15 +CPU= 79.193
  16:02:52 STEP 1538 H= 384:30 +CPU= 70.983
  16:03:58 STEP 1539 H= 384:45 +CPU= 66.288
  16:05:05 STEP 1540 H= 385:00 +CPU= 66.673
  16:06:11 STEP 1541 H= 385:15 +CPU= 65.533
  16:07:19 STEP 1542 H= 385:30 +CPU= 67.489
  16:08:24 STEP 1543 H= 385:45 +CPU= 64.859
  16:09:30 STEP 1544 H= 386:00 +CPU= 66.583
  16:10:36 STEP 1545 H= 386:15 +CPU= 65.144
  16:11:40 STEP 1546 H= 386:30 +CPU= 63.895
  16:12:42 STEP 1547 H= 386:45 +CPU= 62.622
  16:14:01 STEP 1548 H= 387:00 +CPU= 78.848
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001548
  16:15:08 STEP 1549 H= 387:15 +CPU= 66.539
  16:16:16 STEP 1550 H= 387:30 +CPU= 67.852
  16:17:27 STEP 1551 H= 387:45 +CPU= 71.284
  16:18:41 STEP 1552 H= 388:00 +CPU= 73.932
  16:19:51 STEP 1553 H= 388:15 +CPU= 69.660
  16:21:03 STEP 1554 H= 388:30 +CPU= 72.116
  16:22:15 STEP 1555 H= 388:45 +CPU= 72.405
  16:23:31 STEP 1556 H= 389:00 +CPU= 75.946
  16:24:44 STEP 1557 H= 389:15 +CPU= 73.215
  16:25:55 STEP 1558 H= 389:30 +CPU= 70.791
  16:27:09 STEP 1559 H= 389:45 +CPU= 72.906
  16:28:41 STEP 1560 H= 390:00 +CPU= 92.907
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001560
  16:29:55 STEP 1561 H= 390:15 +CPU= 73.298
  16:31:09 STEP 1562 H= 390:30 +CPU= 73.930
  16:32:25 STEP 1563 H= 390:45 +CPU= 75.620
  16:33:42 STEP 1564 H= 391:00 +CPU= 77.807
  16:34:59 STEP 1565 H= 391:15 +CPU= 76.287
  16:36:12 STEP 1566 H= 391:30 +CPU= 72.790
  16:37:25 STEP 1567 H= 391:45 +CPU= 72.929
  16:38:41 STEP 1568 H= 392:00 +CPU= 75.396
  16:39:57 STEP 1569 H= 392:15 +CPU= 76.292
  16:41:10 STEP 1570 H= 392:30 +CPU= 73.000
  16:42:24 STEP 1571 H= 392:45 +CPU= 73.457
  16:43:57 STEP 1572 H= 393:00 +CPU= 92.771
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001572
  16:45:10 STEP 1573 H= 393:15 +CPU= 73.061
  16:46:23 STEP 1574 H= 393:30 +CPU= 72.916
  16:47:33 STEP 1575 H= 393:45 +CPU= 70.815
  16:48:45 STEP 1576 H= 394:00 +CPU= 71.221
  16:49:54 STEP 1577 H= 394:15 +CPU= 69.064
  16:51:03 STEP 1578 H= 394:30 +CPU= 68.739
  16:52:12 STEP 1579 H= 394:45 +CPU= 69.657
  16:53:21 STEP 1580 H= 395:00 +CPU= 69.235
  16:54:32 STEP 1581 H= 395:15 +CPU= 70.643
  16:55:42 STEP 1582 H= 395:30 +CPU= 69.693
  16:56:52 STEP 1583 H= 395:45 +CPU= 70.299
  16:58:28 STEP 1584 H= 396:00 +CPU= 96.103
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001584
  16:59:38 STEP 1585 H= 396:15 +CPU= 68.291
  17:00:48 STEP 1586 H= 396:30 +CPU= 69.185
  17:01:58 STEP 1587 H= 396:45 +CPU= 70.044
  17:03:08 STEP 1588 H= 397:00 +CPU= 70.440
  17:04:19 STEP 1589 H= 397:15 +CPU= 70.769
  17:05:25 STEP 1590 H= 397:30 +CPU= 66.343
  17:06:32 STEP 1591 H= 397:45 +CPU= 66.523
  17:07:45 STEP 1592 H= 398:00 +CPU= 72.348
  17:08:59 STEP 1593 H= 398:15 +CPU= 74.543
  17:10:11 STEP 1594 H= 398:30 +CPU= 72.038
  17:11:23 STEP 1595 H= 398:45 +CPU= 72.077
  17:12:51 STEP 1596 H= 399:00 +CPU= 87.551
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001596
  17:13:59 STEP 1597 H= 399:15 +CPU= 67.582
  17:15:11 STEP 1598 H= 399:30 +CPU= 71.082
  17:16:23 STEP 1599 H= 399:45 +CPU= 71.330
  17:17:38 STEP 1600 H= 400:00 +CPU= 74.971
  17:18:54 STEP 1601 H= 400:15 +CPU= 76.087
  17:20:08 STEP 1602 H= 400:30 +CPU= 73.827
  17:21:19 STEP 1603 H= 400:45 +CPU= 70.460
  17:22:30 STEP 1604 H= 401:00 +CPU= 71.305
  17:23:42 STEP 1605 H= 401:15 +CPU= 71.622
  17:24:58 STEP 1606 H= 401:30 +CPU= 75.387
  17:26:12 STEP 1607 H= 401:45 +CPU= 74.484
  17:27:47 STEP 1608 H= 402:00 +CPU= 93.937
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001608
  17:29:02 STEP 1609 H= 402:15 +CPU= 75.491
  17:30:17 STEP 1610 H= 402:30 +CPU= 74.596
  17:31:32 STEP 1611 H= 402:45 +CPU= 75.622
  17:32:48 STEP 1612 H= 403:00 +CPU= 75.105
  17:34:01 STEP 1613 H= 403:15 +CPU= 72.805
  17:35:13 STEP 1614 H= 403:30 +CPU= 72.086
  17:36:25 STEP 1615 H= 403:45 +CPU= 72.018
  17:37:37 STEP 1616 H= 404:00 +CPU= 71.583
  17:38:48 STEP 1617 H= 404:15 +CPU= 70.636
  17:40:00 STEP 1618 H= 404:30 +CPU= 72.208
  17:41:11 STEP 1619 H= 404:45 +CPU= 71.473
  17:42:43 STEP 1620 H= 405:00 +CPU= 91.527
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001620
  17:43:55 STEP 1621 H= 405:15 +CPU= 71.645
  17:45:06 STEP 1622 H= 405:30 +CPU= 70.910
  17:46:15 STEP 1623 H= 405:45 +CPU= 69.870
  17:47:27 STEP 1624 H= 406:00 +CPU= 71.094
  17:48:38 STEP 1625 H= 406:15 +CPU= 70.753
  17:49:51 STEP 1626 H= 406:30 +CPU= 73.193
  17:51:03 STEP 1627 H= 406:45 +CPU= 72.179
  17:52:20 STEP 1628 H= 407:00 +CPU= 76.475
  17:53:34 STEP 1629 H= 407:15 +CPU= 73.871
  17:54:49 STEP 1630 H= 407:30 +CPU= 74.664
  17:56:02 STEP 1631 H= 407:45 +CPU= 73.202
  17:57:44 STEP 1632 H= 408:00 +CPU=101.182
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2_r391903846_16.zip
Uploading the intermediate file: upload_file_16.zip
  17:59:03 STEP 1633 H= 408:15 +CPU= 78.710
  18:00:23 STEP 1634 H= 408:30 +CPU= 76.688
  18:01:33 STEP 1635 H= 408:45 +CPU= 69.570
  18:02:41 STEP 1636 H= 409:00 +CPU= 68.487
  18:03:46 STEP 1637 H= 409:15 +CPU= 64.636
  18:04:52 STEP 1638 H= 409:30 +CPU= 65.597
  18:05:58 STEP 1639 H= 409:45 +CPU= 65.906
  18:07:06 STEP 1640 H= 410:00 +CPU= 67.956
  18:08:16 STEP 1641 H= 410:15 +CPU= 70.353
  18:09:27 STEP 1642 H= 410:30 +CPU= 70.710
  18:10:35 STEP 1643 H= 410:45 +CPU= 67.856
  18:11:58 STEP 1644 H= 411:00 +CPU= 83.262
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001644
  18:13:02 STEP 1645 H= 411:15 +CPU= 63.634
  18:14:06 STEP 1646 H= 411:30 +CPU= 64.236
  18:15:09 STEP 1647 H= 411:45 +CPU= 62.795
  18:16:17 STEP 1648 H= 412:00 +CPU= 67.636
  18:17:23 STEP 1649 H= 412:15 +CPU= 65.894
  18:18:29 STEP 1650 H= 412:30 +CPU= 65.371
  18:19:36 STEP 1651 H= 412:45 +CPU= 66.959
  18:20:44 STEP 1652 H= 413:00 +CPU= 68.557
  18:21:51 STEP 1653 H= 413:15 +CPU= 66.294
  18:22:57 STEP 1654 H= 413:30 +CPU= 66.448
  18:24:04 STEP 1655 H= 413:45 +CPU= 66.793
  18:25:35 STEP 1656 H= 414:00 +CPU= 90.822
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001656
  18:26:44 STEP 1657 H= 414:15 +CPU= 69.776
  18:27:56 STEP 1658 H= 414:30 +CPU= 71.380
  18:29:05 STEP 1659 H= 414:45 +CPU= 69.273
  18:30:14 STEP 1660 H= 415:00 +CPU= 68.313
  18:31:20 STEP 1661 H= 415:15 +CPU= 66.745
  18:32:27 STEP 1662 H= 415:30 +CPU= 66.705
  18:33:35 STEP 1663 H= 415:45 +CPU= 67.782
  18:34:44 STEP 1664 H= 416:00 +CPU= 69.044
  18:35:49 STEP 1665 H= 416:15 +CPU= 65.294
  18:36:59 STEP 1666 H= 416:30 +CPU= 69.288
  18:38:10 STEP 1667 H= 416:45 +CPU= 71.709
  18:39:39 STEP 1668 H= 417:00 +CPU= 88.263
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001668
  18:40:51 STEP 1669 H= 417:15 +CPU= 72.300
  18:42:00 STEP 1670 H= 417:30 +CPU= 68.401
  18:43:11 STEP 1671 H= 417:45 +CPU= 70.846
  18:44:22 STEP 1672 H= 418:00 +CPU= 70.621
  18:45:27 STEP 1673 H= 418:15 +CPU= 65.417
  18:46:33 STEP 1674 H= 418:30 +CPU= 65.256
  18:47:40 STEP 1675 H= 418:45 +CPU= 66.783
  18:48:47 STEP 1676 H= 419:00 +CPU= 66.939
  18:49:53 STEP 1677 H= 419:15 +CPU= 66.185
  18:50:44 STEP 1678 H= 419:30 +CPU= 51.585
  18:51:51 STEP 1679 H= 419:45 +CPU= 66.633
  18:53:26 STEP 1680 H= 420:00 +CPU= 93.986
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001680
  18:54:34 STEP 1681 H= 420:15 +CPU= 68.234
  18:55:39 STEP 1682 H= 420:30 +CPU= 64.418
  18:56:43 STEP 1683 H= 420:45 +CPU= 64.185
  18:57:50 STEP 1684 H= 421:00 +CPU= 67.499
  18:58:55 STEP 1685 H= 421:15 +CPU= 64.883
  19:00:00 STEP 1686 H= 421:30 +CPU= 64.095
  19:01:08 STEP 1687 H= 421:45 +CPU= 67.375
  19:02:20 STEP 1688 H= 422:00 +CPU= 71.389
  19:03:31 STEP 1689 H= 422:15 +CPU= 70.418
  19:04:39 STEP 1690 H= 422:30 +CPU= 68.481
  19:05:43 STEP 1691 H= 422:45 +CPU= 63.380
  19:07:03 STEP 1692 H= 423:00 +CPU= 80.081
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001692
  19:08:12 STEP 1693 H= 423:15 +CPU= 68.810
  19:09:21 STEP 1694 H= 423:30 +CPU= 69.304
  19:10:27 STEP 1695 H= 423:45 +CPU= 66.205
  19:11:37 STEP 1696 H= 424:00 +CPU= 69.922
  19:12:48 STEP 1697 H= 424:15 +CPU= 70.968
  19:13:57 STEP 1698 H= 424:30 +CPU= 69.048
  19:15:07 STEP 1699 H= 424:45 +CPU= 69.429
  19:16:17 STEP 1700 H= 425:00 +CPU= 69.538
  19:17:23 STEP 1701 H= 425:15 +CPU= 65.594
  19:18:38 STEP 1702 H= 425:30 +CPU= 74.527
  19:19:48 STEP 1703 H= 425:45 +CPU= 69.873
  19:21:10 STEP 1704 H= 426:00 +CPU= 81.923
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001704
  19:22:15 STEP 1705 H= 426:15 +CPU= 65.424
  19:23:20 STEP 1706 H= 426:30 +CPU= 64.798
  19:24:26 STEP 1707 H= 426:45 +CPU= 66.083
  19:25:38 STEP 1708 H= 427:00 +CPU= 72.329
  19:26:50 STEP 1709 H= 427:15 +CPU= 71.312
  19:28:01 STEP 1710 H= 427:30 +CPU= 71.062
  19:29:12 STEP 1711 H= 427:45 +CPU= 71.279
  19:30:24 STEP 1712 H= 428:00 +CPU= 71.476
  19:31:36 STEP 1713 H= 428:15 +CPU= 72.060
  19:32:50 STEP 1714 H= 428:30 +CPU= 74.344
  19:34:03 STEP 1715 H= 428:45 +CPU= 72.824
  19:35:32 STEP 1716 H= 429:00 +CPU= 88.987
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001716
  19:36:44 STEP 1717 H= 429:15 +CPU= 71.423
  19:37:55 STEP 1718 H= 429:30 +CPU= 70.651
  19:39:05 STEP 1719 H= 429:45 +CPU= 69.857
  19:40:16 STEP 1720 H= 430:00 +CPU= 70.944
  19:41:26 STEP 1721 H= 430:15 +CPU= 70.651
  19:42:38 STEP 1722 H= 430:30 +CPU= 71.348
  19:43:49 STEP 1723 H= 430:45 +CPU= 70.761
  19:45:01 STEP 1724 H= 431:00 +CPU= 71.689
  19:46:12 STEP 1725 H= 431:15 +CPU= 71.338
  19:47:21 STEP 1726 H= 431:30 +CPU= 68.685
  19:48:29 STEP 1727 H= 431:45 +CPU= 67.714
  19:50:05 STEP 1728 H= 432:00 +CPU= 95.974
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290011/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2_r391903846_17.zip
Uploading the intermediate file: upload_file_17.zip
  19:51:23 STEP 1729 H= 432:15 +CPU= 76.762
  19:52:44 STEP 1730 H= 432:30 +CPU= 80.415
  19:53:55 STEP 1731 H= 432:45 +CPU= 70.374
  19:55:03 STEP 1732 H= 433:00 +CPU= 67.806
  19:56:09 STEP 1733 H= 433:15 +CPU= 66.029
  19:57:16 STEP 1734 H= 433:30 +CPU= 67.179
  19:58:22 STEP 1735 H= 433:45 +CPU= 66.513
  19:59:28 STEP 1736 H= 434:00 +CPU= 65.842
  20:00:34 STEP 1737 H= 434:15 +CPU= 65.198
  20:01:43 STEP 1738 H= 434:30 +CPU= 68.060
  20:02:51 STEP 1739 H= 434:45 +CPU= 67.736
  20:04:14 STEP 1740 H= 435:00 +CPU= 83.176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001740
  20:05:19 STEP 1741 H= 435:15 +CPU= 65.004
  20:06:24 STEP 1742 H= 435:30 +CPU= 64.983
  20:07:29 STEP 1743 H= 435:45 +CPU= 64.659
  20:08:37 STEP 1744 H= 436:00 +CPU= 68.336
  20:09:46 STEP 1745 H= 436:15 +CPU= 68.665
  20:10:54 STEP 1746 H= 436:30 +CPU= 68.320
  20:12:05 STEP 1747 H= 436:45 +CPU= 70.385
  20:13:16 STEP 1748 H= 437:00 +CPU= 71.688
  20:14:26 STEP 1749 H= 437:15 +CPU= 69.650
  20:15:35 STEP 1750 H= 437:30 +CPU= 69.153
  20:16:40 STEP 1751 H= 437:45 +CPU= 64.979
  20:18:03 STEP 1752 H= 438:00 +CPU= 82.446
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001752
  20:19:09 STEP 1753 H= 438:15 +CPU= 65.945
  20:20:17 STEP 1754 H= 438:30 +CPU= 67.989
  20:21:27 STEP 1755 H= 438:45 +CPU= 69.770
  20:22:38 STEP 1756 H= 439:00 +CPU= 70.845
  20:23:46 STEP 1757 H= 439:15 +CPU= 68.011
  20:24:53 STEP 1758 H= 439:30 +CPU= 67.675
  20:26:02 STEP 1759 H= 439:45 +CPU= 69.029
  20:27:14 STEP 1760 H= 440:00 +CPU= 70.992
  20:28:29 STEP 1761 H= 440:15 +CPU= 75.455
  20:29:53 STEP 1762 H= 440:30 +CPU= 83.445
  20:31:05 STEP 1763 H= 440:45 +CPU= 71.290
  20:32:33 STEP 1764 H= 441:00 +CPU= 88.486
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001764
  20:33:42 STEP 1765 H= 441:15 +CPU= 68.592
  20:34:50 STEP 1766 H= 441:30 +CPU= 67.842
  20:36:00 STEP 1767 H= 441:45 +CPU= 69.785
  20:37:12 STEP 1768 H= 442:00 +CPU= 72.254
  20:38:23 STEP 1769 H= 442:15 +CPU= 71.290
  20:39:34 STEP 1770 H= 442:30 +CPU= 71.141
  20:40:48 STEP 1771 H= 442:45 +CPU= 73.971
  20:42:01 STEP 1772 H= 443:00 +CPU= 73.153
  20:43:13 STEP 1773 H= 443:15 +CPU= 71.176
  20:44:23 STEP 1774 H= 443:30 +CPU= 70.372
  20:45:32 STEP 1775 H= 443:45 +CPU= 69.173
  20:47:06 STEP 1776 H= 444:00 +CPU= 94.030
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001776
  20:48:16 STEP 1777 H= 444:15 +CPU= 69.462
  20:49:24 STEP 1778 H= 444:30 +CPU= 68.222
  20:50:29 STEP 1779 H= 444:45 +CPU= 64.818
  20:51:36 STEP 1780 H= 445:00 +CPU= 67.539
  20:52:44 STEP 1781 H= 445:15 +CPU= 66.969
  20:53:52 STEP 1782 H= 445:30 +CPU= 67.682
  20:54:59 STEP 1783 H= 445:45 +CPU= 66.975
  20:56:10 STEP 1784 H= 446:00 +CPU= 70.884
  20:57:20 STEP 1785 H= 446:15 +CPU= 70.077
  20:58:30 STEP 1786 H= 446:30 +CPU= 70.454
  20:59:38 STEP 1787 H= 446:45 +CPU= 68.049
  21:01:09 STEP 1788 H= 447:00 +CPU= 90.627
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001788
  21:02:24 STEP 1789 H= 447:15 +CPU= 74.894
  21:03:39 STEP 1790 H= 447:30 +CPU= 75.003
  21:04:52 STEP 1791 H= 447:45 +CPU= 72.924
  21:06:08 STEP 1792 H= 448:00 +CPU= 76.194
  21:07:24 STEP 1793 H= 448:15 +CPU= 75.321
  21:08:37 STEP 1794 H= 448:30 +CPU= 72.757
  21:09:50 STEP 1795 H= 448:45 +CPU= 73.263
  21:11:03 STEP 1796 H= 449:00 +CPU= 72.785
  21:12:25 STEP 1797 H= 449:15 +CPU= 81.870
  21:13:35 STEP 1798 H= 449:30 +CPU= 70.220
  21:14:48 STEP 1799 H= 449:45 +CPU= 72.599
  21:16:21 STEP 1800 H= 450:00 +CPU= 91.819
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001800
  21:17:36 STEP 1801 H= 450:15 +CPU= 75.061
  21:18:50 STEP 1802 H= 450:30 +CPU= 73.885
  21:20:05 STEP 1803 H= 450:45 +CPU= 74.957
  21:21:19 STEP 1804 H= 451:00 +CPU= 74.207
  21:22:33 STEP 1805 H= 451:15 +CPU= 73.991
  21:23:47 STEP 1806 H= 451:30 +CPU= 73.990
  21:25:01 STEP 1807 H= 451:45 +CPU= 73.692
  21:26:16 STEP 1808 H= 452:00 +CPU= 74.751
  21:27:27 STEP 1809 H= 452:15 +CPU= 71.217
  21:28:36 STEP 1810 H= 452:30 +CPU= 68.767
  21:29:48 STEP 1811 H= 452:45 +CPU= 72.129
  21:31:18 STEP 1812 H= 453:00 +CPU= 90.349
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAh7zg+001812
  21:32:28 STEP 1813 H= 453:15 +CPU= 69.887
  21:33:41 STEP 1814 H= 453:30 +CPU= 72.602
  21:34:51 STEP 1815 H= 453:45 +CPU= 70.226
  21:36:03 STEP 1816 H= 454:00 +CPU= 71.759
  21:37:14 STEP 1817 H= 454:15 +CPU= 70.588
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.768] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 3990MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.768] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3990 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CPG_DRV 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.769] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-CPG                  (1025) 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.770] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              CPG 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.770] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CPG_DIA 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213722:1718599042:127784.770] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                GPRH 
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/4/oifs_43r3_model.exe' : sigcontextptr=0x7ffc4bb01650
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 22) with plain hex-dump
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x83a8f3] : 0x83a8f3
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x83aef9] : 0x83aef9
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x1dd5490] : 0x1dd5490
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x1cc6692] : 0x1cc6692
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0xcc6be6] : 0xcc6be6
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0xcbe7ec] : 0xcbe7ec
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0xc2519f] : 0xc2519f
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0xc22238] : 0xc22238
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0xbcc914] : 0xbcc914
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x12c5002] : 0x12c5002
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0xbacbae] : 0xbacbae
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0x688153] : 0x688153
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x4cd1dc] : 0x4cd1dc
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x4b5d85] : 0x4b5d85
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x4b4f52] : 0x4b4f52
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x49d638] : 0x49d638
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [16]: [0x406fbd] : 0x406fbd
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [17]: [0x401e7f] : 0x401e7f
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [18]: [0x401e12] : 0x401e12
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [19]: [0x1dcc34a] : 0x1dcc34a
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [20]: [0x1dcdbe7] : 0x1dcdbe7
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [21]: [0x401cd5] : 0x401cd5
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.770] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/4/oifs_43r3_model.exe' : sigcontextptr=0x7ffc4bb01560
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 23) with plain hex-dump
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x83a8f3] : 0x83a8f3
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x83aef9] : 0x83aef9
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x822540] : 0x822540
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x1dd5490] : 0x1dd5490
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0x1cc6692] : 0x1cc6692
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0xcc6be6] : 0xcc6be6
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0xcbe7ec] : 0xcbe7ec
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0xc2519f] : 0xc2519f
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0xc22238] : 0xc22238
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0xbcc914] : 0xbcc914
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0x12c5002] : 0x12c5002
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0xbacbae] : 0xbacbae
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x688153] : 0x688153
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x4cd1dc] : 0x4cd1dc
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x4b5d85] : 0x4b5d85
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x4b4f52] : 0x4b4f52
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [16]: [0x49d638] : 0x49d638
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [17]: [0x406fbd] : 0x406fbd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [18]: [0x401e7f] : 0x401e7f
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [19]: [0x401e12] : 0x401e12
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [20]: [0x1dcc34a] : 0x1dcc34a
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [21]: [0x1dcdbe7] : 0x1dcdbe7
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:340] [22]: [0x401cd5] : 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.849] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:OpVM:1:1:44734:44734] [20240616:213723:1718599043:127785.849] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000CC6BE6  gprh_                     128  gprh.F90
oifs_43r3_model.e  0000000000CBE7EC  cpg_dia_                  600  cpg_dia.F90
oifs_43r3_model.e  0000000000C2519F  cpg_                      718  cpg.F90
oifs_43r3_model.e  0000000000C22238  cpg_drv_                  351  cpg_drv.F90
oifs_43r3_model.e  0000000000BCC914  gp_model_                 419  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.530954564091575E-05 0.000000000000000E+00 0.117486225665079E-01
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.257920520187520E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.413260611395059E-05 0.000000000000000E+00 0.111947795757522E-01
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.219808856532636E-04 0.000000000000000E+00 0.154377019241466E-01
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.258002730884665E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.180445437604626E-04 0.000000000000000E+00 0.340760784894281E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.140461522019741E-04 0.000000000000000E+00 0.692988216826612E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.340928029324356E-05 0.000000000000000E+00 0.153742341040026E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.555046043635921E-06 0.000000000000000E+00 0.889258033193135E-02
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.244334615721858E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1812 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1812 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001812 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001812
 MPI-TASK:   1 -    306865316 BYTES IN      6 RECORDS TRANSFERRED IN    0.0010 SECONDS306865.3 Mbytes/s, TOTAL TIME=    0.0339( 2.9%)
 NSTEP =  1812 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001812 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001812
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0292 SECONDS 24226.0 Mbytes/s, TOTAL TIME=    1.0982( 2.7%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001812 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001812
 MPI-TASK:   1 -   4909845056 BYTES IN    125 RECORDS TRANSFERRED IN    0.0206 SECONDS238342.0 Mbytes/s, TOTAL TIME=    0.7938( 2.6%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001812 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001812
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0003 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0105( 2.9%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001812 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001812
 MPI-TASK:   1 -    298285587 BYTES IN      4 RECORDS TRANSFERRED IN    0.0005 SECONDS596571.2 Mbytes/s, TOTAL TIME=    0.0219( 2.3%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1812 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1812 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001812 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001812
 MPI-TASK:   1 -    306865316 BYTES IN      7 RECORDS TRANSFERRED IN    0.0017 SECONDS180509.0 Mbytes/s, TOTAL TIME=    0.0412( 4.1%)
 21:31:18 STEP 1812 H= 453:00 +CPU= 90.349
 NSTEP =  1813 STEPO   0AAA00AAA
 21:32:28 STEP 1813 H= 453:15 +CPU= 69.887
 NSTEP =  1814 STEPO   0AAA00AAA
 21:33:41 STEP 1814 H= 453:30 +CPU= 72.602
ISTEP=    1815 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1815 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1815 STEPO   0AAA00AAA
 21:34:51 STEP 1815 H= 453:45 +CPU= 70.226
 NSTEP =  1816 STEPO   0AAA00AAA
 21:36:03 STEP 1816 H= 454:00 +CPU= 71.759
 NSTEP =  1817 STEPO   0AAA00AAA
 21:37:14 STEP 1817 H= 454:15 +CPU= 70.588
 NSTEP =  1818 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 21:30:05 0AAA00AAA STEPO     1812     1.955    1.955    1.956   2097:05   2122:27 0.33694264138714E-04       2GB       0MB
 21:31:18 0AAA00AAA STEPO     1813    73.247   73.247   73.281   2098:18   2123:40 0.33694264138714E-04       2GB       0MB
 21:32:28 0AAA00AAA STEPO     1814    69.887   69.887   69.998   2099:28   2124:50 0.33694264138714E-04       2GB       0MB
 21:33:41 0AAA00AAA STEPO     1815    72.602   72.602   72.609   2100:40   2126:03 0.33694264138714E-04       2GB       0MB
 21:34:51 0AAA00AAA STEPO     1816    70.226   70.226   70.231   2101:51   2127:13 0.33694264138714E-04       2GB       0MB
 21:36:03 0AAA00AAA STEPO     1817    71.760   71.760   71.835   2103:02   2128:25 0.33694264138714E-04       2GB       0MB
 21:37:14 0AAA00AAA STEPO     1818    70.588   70.588   70.601   2104:13   2129:36 0.33694264138714E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1776",
 CTIME   = "00181200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1764, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/4/progress_file_12290011.xml

<running_values>
  <last_cpu_time>126256.590000</last_cpu_time>
  <upload_file_number>18</upload_file_number>
  <last_iter>1818</last_iter>
  <last_upload>1555200</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 Jun 2024 12:25:09 1536326 22449189 oifs_43r3_bl_a1as_2016092300_20_1018_12290011_2 864,000 67,918 0.0786


©2024 climateprediction.net