climateprediction.net home page
Task 22443661

Task 22443661

Name oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0
Workunit 12290072
Created 12 Jun 2024, 22:29:13 UTC
Sent 13 Jun 2024, 21:55:19 UTC
Report deadline 12 Aug 2024, 21:55:19 UTC
Received 23 Jun 2024, 21:48:39 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1551031
Run time 17 hours 59 min 40 sec
CPU time 15 hours 33 min 46 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 4.04 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.18 MB
Peak swap size 5,981.12 MB
Peak disk usage 1,373.89 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
 1497 H= 374:15 +CPU= 19.246
  06:47:46 STEP 1498 H= 374:30 +CPU= 19.497
  06:48:05 STEP 1499 H= 374:45 +CPU= 19.653
  06:48:30 STEP 1500 H= 375:00 +CPU= 24.365
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001500
  06:48:49 STEP 1501 H= 375:15 +CPU= 19.722
  06:49:09 STEP 1502 H= 375:30 +CPU= 19.670
  06:49:29 STEP 1503 H= 375:45 +CPU= 19.649
  06:49:49 STEP 1504 H= 376:00 +CPU= 20.029
  06:50:08 STEP 1505 H= 376:15 +CPU= 19.548
  06:50:28 STEP 1506 H= 376:30 +CPU= 19.700
  06:50:48 STEP 1507 H= 376:45 +CPU= 19.465
  06:51:07 STEP 1508 H= 377:00 +CPU= 19.655
  06:51:27 STEP 1509 H= 377:15 +CPU= 19.528
  06:51:46 STEP 1510 H= 377:30 +CPU= 19.269
  06:52:05 STEP 1511 H= 377:45 +CPU= 19.200
  06:52:29 STEP 1512 H= 378:00 +CPU= 23.954
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001512
  06:52:49 STEP 1513 H= 378:15 +CPU= 19.385
  06:53:08 STEP 1514 H= 378:30 +CPU= 19.654
  06:53:28 STEP 1515 H= 378:45 +CPU= 19.597
  06:53:48 STEP 1516 H= 379:00 +CPU= 20.059
  06:54:08 STEP 1517 H= 379:15 +CPU= 20.023
  06:54:28 STEP 1518 H= 379:30 +CPU= 19.680
  06:54:47 STEP 1519 H= 379:45 +CPU= 19.940
  06:55:08 STEP 1520 H= 380:00 +CPU= 20.139
  06:55:27 STEP 1521 H= 380:15 +CPU= 19.601
  06:55:46 STEP 1522 H= 380:30 +CPU= 19.170
  06:56:06 STEP 1523 H= 380:45 +CPU= 19.292
  06:56:30 STEP 1524 H= 381:00 +CPU= 23.881
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001524
  06:56:49 STEP 1525 H= 381:15 +CPU= 19.250
  06:57:08 STEP 1526 H= 381:30 +CPU= 19.577
  06:57:28 STEP 1527 H= 381:45 +CPU= 19.664
  06:57:48 STEP 1528 H= 382:00 +CPU= 19.586
  06:58:08 STEP 1529 H= 382:15 +CPU= 19.928
  06:58:27 STEP 1530 H= 382:30 +CPU= 19.880
  06:58:47 STEP 1531 H= 382:45 +CPU= 19.708
  06:59:07 STEP 1532 H= 383:00 +CPU= 20.256
  06:59:27 STEP 1533 H= 383:15 +CPU= 19.637
  06:59:47 STEP 1534 H= 383:30 +CPU= 19.503
  07:00:06 STEP 1535 H= 383:45 +CPU= 19.844
  07:00:34 STEP 1536 H= 384:00 +CPU= 27.160
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0_r1487970834_15.zip
Uploading the intermediate file: upload_file_15.zip
  07:00:54 STEP 1537 H= 384:15 +CPU= 20.152
  07:01:13 STEP 1538 H= 384:30 +CPU= 19.688
  07:01:34 STEP 1539 H= 384:45 +CPU= 20.444
  07:01:56 STEP 1540 H= 385:00 +CPU= 21.618
  07:02:15 STEP 1541 H= 385:15 +CPU= 19.672
  07:02:35 STEP 1542 H= 385:30 +CPU= 19.380
  07:02:54 STEP 1543 H= 385:45 +CPU= 19.371
  07:03:14 STEP 1544 H= 386:00 +CPU= 20.194
  07:03:34 STEP 1545 H= 386:15 +CPU= 19.970
  07:03:54 STEP 1546 H= 386:30 +CPU= 20.247
  07:04:15 STEP 1547 H= 386:45 +CPU= 20.086
  07:04:39 STEP 1548 H= 387:00 +CPU= 24.128
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001548
  07:04:58 STEP 1549 H= 387:15 +CPU= 19.279
  07:05:17 STEP 1550 H= 387:30 +CPU= 19.490
  07:05:38 STEP 1551 H= 387:45 +CPU= 20.019
  07:05:58 STEP 1552 H= 388:00 +CPU= 20.346
  07:06:18 STEP 1553 H= 388:15 +CPU= 20.096
  07:06:38 STEP 1554 H= 388:30 +CPU= 19.626
  07:06:57 STEP 1555 H= 388:45 +CPU= 19.771
  07:07:17 STEP 1556 H= 389:00 +CPU= 19.637
  07:07:36 STEP 1557 H= 389:15 +CPU= 19.347
  07:07:56 STEP 1558 H= 389:30 +CPU= 19.575
  07:08:15 STEP 1559 H= 389:45 +CPU= 19.368
  07:08:39 STEP 1560 H= 390:00 +CPU= 24.054
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001560
  07:08:59 STEP 1561 H= 390:15 +CPU= 19.274
  07:09:18 STEP 1562 H= 390:30 +CPU= 19.319
  07:09:37 STEP 1563 H= 390:45 +CPU= 19.257
  07:09:57 STEP 1564 H= 391:00 +CPU= 20.106
  07:10:18 STEP 1565 H= 391:15 +CPU= 20.230
  07:10:37 STEP 1566 H= 391:30 +CPU= 19.396
  07:10:56 STEP 1567 H= 391:45 +CPU= 19.318
  07:11:16 STEP 1568 H= 392:00 +CPU= 19.769
  07:11:35 STEP 1569 H= 392:15 +CPU= 19.334
  07:11:55 STEP 1570 H= 392:30 +CPU= 19.485
  07:12:14 STEP 1571 H= 392:45 +CPU= 19.461
  07:12:39 STEP 1572 H= 393:00 +CPU= 24.227
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001572
  07:12:58 STEP 1573 H= 393:15 +CPU= 19.785
  07:13:18 STEP 1574 H= 393:30 +CPU= 19.819
  07:13:37 STEP 1575 H= 393:45 +CPU= 19.255
  07:13:57 STEP 1576 H= 394:00 +CPU= 19.549
  07:14:17 STEP 1577 H= 394:15 +CPU= 19.556
  07:14:36 STEP 1578 H= 394:30 +CPU= 19.337
  07:14:55 STEP 1579 H= 394:45 +CPU= 19.374
  07:15:15 STEP 1580 H= 395:00 +CPU= 19.723
  07:15:34 STEP 1581 H= 395:15 +CPU= 19.327
  07:15:54 STEP 1582 H= 395:30 +CPU= 19.355
  07:16:13 STEP 1583 H= 395:45 +CPU= 19.349
  07:16:40 STEP 1584 H= 396:00 +CPU= 26.813
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001584
  07:16:59 STEP 1585 H= 396:15 +CPU= 19.434
  07:17:19 STEP 1586 H= 396:30 +CPU= 19.537
  07:17:38 STEP 1587 H= 396:45 +CPU= 19.544
  07:17:59 STEP 1588 H= 397:00 +CPU= 20.555
  07:18:18 STEP 1589 H= 397:15 +CPU= 19.393
  07:18:38 STEP 1590 H= 397:30 +CPU= 19.353
  07:18:57 STEP 1591 H= 397:45 +CPU= 19.541
  07:19:17 STEP 1592 H= 398:00 +CPU= 19.610
  07:19:36 STEP 1593 H= 398:15 +CPU= 19.210
  07:19:55 STEP 1594 H= 398:30 +CPU= 19.245
  07:20:15 STEP 1595 H= 398:45 +CPU= 19.503
  07:20:39 STEP 1596 H= 399:00 +CPU= 24.239
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001596
  07:20:58 STEP 1597 H= 399:15 +CPU= 19.235
  07:21:18 STEP 1598 H= 399:30 +CPU= 19.293
  07:21:37 STEP 1599 H= 399:45 +CPU= 19.278
  07:21:56 STEP 1600 H= 400:00 +CPU= 19.622
  07:22:16 STEP 1601 H= 400:15 +CPU= 19.257
  07:22:35 STEP 1602 H= 400:30 +CPU= 19.246
  07:22:54 STEP 1603 H= 400:45 +CPU= 19.308
  07:23:14 STEP 1604 H= 401:00 +CPU= 19.625
  07:23:33 STEP 1605 H= 401:15 +CPU= 19.317
  07:23:53 STEP 1606 H= 401:30 +CPU= 19.716
  07:24:13 STEP 1607 H= 401:45 +CPU= 19.798
  07:24:37 STEP 1608 H= 402:00 +CPU= 24.180
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001608
  07:24:57 STEP 1609 H= 402:15 +CPU= 19.641
  07:25:16 STEP 1610 H= 402:30 +CPU= 19.677
  07:25:36 STEP 1611 H= 402:45 +CPU= 19.739
  07:25:56 STEP 1612 H= 403:00 +CPU= 20.290
  07:26:16 STEP 1613 H= 403:15 +CPU= 19.734
  07:26:36 STEP 1614 H= 403:30 +CPU= 19.649
  07:26:55 STEP 1615 H= 403:45 +CPU= 19.224
  07:27:15 STEP 1616 H= 404:00 +CPU= 19.709
  07:27:34 STEP 1617 H= 404:15 +CPU= 19.477
  07:27:53 STEP 1618 H= 404:30 +CPU= 19.289
  07:28:13 STEP 1619 H= 404:45 +CPU= 19.234
  07:28:37 STEP 1620 H= 405:00 +CPU= 23.922
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001620
  07:28:56 STEP 1621 H= 405:15 +CPU= 19.168
  07:29:15 STEP 1622 H= 405:30 +CPU= 19.179
  07:29:34 STEP 1623 H= 405:45 +CPU= 19.197
  07:29:54 STEP 1624 H= 406:00 +CPU= 19.644
  07:30:13 STEP 1625 H= 406:15 +CPU= 19.731
  07:30:33 STEP 1626 H= 406:30 +CPU= 19.757
  07:30:53 STEP 1627 H= 406:45 +CPU= 19.479
  07:31:13 STEP 1628 H= 407:00 +CPU= 20.148
  07:31:33 STEP 1629 H= 407:15 +CPU= 19.686
  07:31:52 STEP 1630 H= 407:30 +CPU= 19.806
  07:32:12 STEP 1631 H= 407:45 +CPU= 19.726
  07:32:39 STEP 1632 H= 408:00 +CPU= 26.651
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0_r1487970834_16.zip
Uploading the intermediate file: upload_file_16.zip
  07:32:58 STEP 1633 H= 408:15 +CPU= 19.751
  07:33:19 STEP 1634 H= 408:30 +CPU= 20.151
  07:33:38 STEP 1635 H= 408:45 +CPU= 19.602
  07:33:59 STEP 1636 H= 409:00 +CPU= 20.430
  07:34:18 STEP 1637 H= 409:15 +CPU= 19.572
  07:34:38 STEP 1638 H= 409:30 +CPU= 19.474
  07:34:57 STEP 1639 H= 409:45 +CPU= 19.525
  07:35:17 STEP 1640 H= 410:00 +CPU= 19.835
  07:35:37 STEP 1641 H= 410:15 +CPU= 19.479
  07:35:56 STEP 1642 H= 410:30 +CPU= 19.595
  07:36:16 STEP 1643 H= 410:45 +CPU= 19.594
  07:36:40 STEP 1644 H= 411:00 +CPU= 24.462
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001644
  07:37:00 STEP 1645 H= 411:15 +CPU= 19.752
  07:37:20 STEP 1646 H= 411:30 +CPU= 19.731
  07:37:39 STEP 1647 H= 411:45 +CPU= 19.752
  07:38:00 STEP 1648 H= 412:00 +CPU= 20.264
  07:38:19 STEP 1649 H= 412:15 +CPU= 19.426
  07:38:39 STEP 1650 H= 412:30 +CPU= 19.480
  07:38:58 STEP 1651 H= 412:45 +CPU= 19.502
  07:39:18 STEP 1652 H= 413:00 +CPU= 19.830
  07:39:37 STEP 1653 H= 413:15 +CPU= 19.459
  07:39:57 STEP 1654 H= 413:30 +CPU= 19.500
  07:40:17 STEP 1655 H= 413:45 +CPU= 19.664
  07:40:41 STEP 1656 H= 414:00 +CPU= 24.227
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001656
  07:41:00 STEP 1657 H= 414:15 +CPU= 19.313
  07:41:19 STEP 1658 H= 414:30 +CPU= 19.102
  07:41:38 STEP 1659 H= 414:45 +CPU= 19.163
  07:41:58 STEP 1660 H= 415:00 +CPU= 19.574
  07:42:17 STEP 1661 H= 415:15 +CPU= 19.251
  07:42:36 STEP 1662 H= 415:30 +CPU= 19.216
  07:42:56 STEP 1663 H= 415:45 +CPU= 19.136
  07:43:15 STEP 1664 H= 416:00 +CPU= 19.515
  07:43:34 STEP 1665 H= 416:15 +CPU= 19.145
  07:43:53 STEP 1666 H= 416:30 +CPU= 19.146
  07:44:13 STEP 1667 H= 416:45 +CPU= 19.148
  07:44:37 STEP 1668 H= 417:00 +CPU= 24.258
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001668
  07:44:57 STEP 1669 H= 417:15 +CPU= 19.787
  07:45:16 STEP 1670 H= 417:30 +CPU= 19.350
  07:45:36 STEP 1671 H= 417:45 +CPU= 19.691
  07:45:56 STEP 1672 H= 418:00 +CPU= 20.123
  07:46:15 STEP 1673 H= 418:15 +CPU= 19.604
  07:46:35 STEP 1674 H= 418:30 +CPU= 19.479
  07:46:54 STEP 1675 H= 418:45 +CPU= 19.504
  07:47:14 STEP 1676 H= 419:00 +CPU= 20.095
  07:47:34 STEP 1677 H= 419:15 +CPU= 19.674
  07:47:54 STEP 1678 H= 419:30 +CPU= 19.699
  07:48:14 STEP 1679 H= 419:45 +CPU= 19.780
  07:48:40 STEP 1680 H= 420:00 +CPU= 26.836
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001680
  07:49:00 STEP 1681 H= 420:15 +CPU= 19.313
  07:49:20 STEP 1682 H= 420:30 +CPU= 19.824
  07:49:39 STEP 1683 H= 420:45 +CPU= 19.694
  07:49:59 STEP 1684 H= 421:00 +CPU= 19.930
  07:50:19 STEP 1685 H= 421:15 +CPU= 19.503
  07:50:38 STEP 1686 H= 421:30 +CPU= 19.522
  07:50:58 STEP 1687 H= 421:45 +CPU= 19.509
  07:51:18 STEP 1688 H= 422:00 +CPU= 19.881
  07:51:37 STEP 1689 H= 422:15 +CPU= 19.507
  07:51:57 STEP 1690 H= 422:30 +CPU= 19.528
  07:52:16 STEP 1691 H= 422:45 +CPU= 19.502
  07:52:40 STEP 1692 H= 423:00 +CPU= 24.298
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001692
  07:53:00 STEP 1693 H= 423:15 +CPU= 19.554
  07:53:20 STEP 1694 H= 423:30 +CPU= 19.554
  07:53:39 STEP 1695 H= 423:45 +CPU= 19.541
  07:53:59 STEP 1696 H= 424:00 +CPU= 19.940
  07:54:19 STEP 1697 H= 424:15 +CPU= 19.579
  07:54:38 STEP 1698 H= 424:30 +CPU= 19.625
  07:54:58 STEP 1699 H= 424:45 +CPU= 19.359
  07:55:17 STEP 1700 H= 425:00 +CPU= 19.840
  07:55:37 STEP 1701 H= 425:15 +CPU= 19.688
  07:55:57 STEP 1702 H= 425:30 +CPU= 19.851
  07:56:17 STEP 1703 H= 425:45 +CPU= 19.710
  07:56:42 STEP 1704 H= 426:00 +CPU= 24.793
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001704
  07:57:02 STEP 1705 H= 426:15 +CPU= 20.184
  07:57:22 STEP 1706 H= 426:30 +CPU= 19.854
  07:57:41 STEP 1707 H= 426:45 +CPU= 19.860
  07:58:01 STEP 1708 H= 427:00 +CPU= 19.816
  07:58:21 STEP 1709 H= 427:15 +CPU= 19.943
  07:58:41 STEP 1710 H= 427:30 +CPU= 19.891
  07:59:01 STEP 1711 H= 427:45 +CPU= 19.726
  07:59:21 STEP 1712 H= 428:00 +CPU= 20.404
  07:59:41 STEP 1713 H= 428:15 +CPU= 19.582
  08:00:00 STEP 1714 H= 428:30 +CPU= 19.324
  08:00:19 STEP 1715 H= 428:45 +CPU= 19.153
  08:00:43 STEP 1716 H= 429:00 +CPU= 23.995
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001716
  08:01:03 STEP 1717 H= 429:15 +CPU= 19.479
  08:01:22 STEP 1718 H= 429:30 +CPU= 19.417
  08:01:42 STEP 1719 H= 429:45 +CPU= 19.693
  08:02:02 STEP 1720 H= 430:00 +CPU= 20.082
  08:02:21 STEP 1721 H= 430:15 +CPU= 19.375
  08:02:41 STEP 1722 H= 430:30 +CPU= 19.253
  08:03:00 STEP 1723 H= 430:45 +CPU= 19.319
  08:03:20 STEP 1724 H= 431:00 +CPU= 19.690
  08:03:39 STEP 1725 H= 431:15 +CPU= 19.444
  08:03:58 STEP 1726 H= 431:30 +CPU= 19.425
  08:04:18 STEP 1727 H= 431:45 +CPU= 19.331
  08:04:45 STEP 1728 H= 432:00 +CPU= 26.982
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0_r1487970834_17.zip
Uploading the intermediate file: upload_file_17.zip
  08:05:05 STEP 1729 H= 432:15 +CPU= 20.454
  08:05:24 STEP 1730 H= 432:30 +CPU= 19.218
  08:05:46 STEP 1731 H= 432:45 +CPU= 20.890
Quit request received from BOINC client, ending the child process
>>> Printing last 70 lines from file: NODE.001_01

 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.255752086661613E-05 0.000000000000000E+00 0.840943159175486E-03
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.236545235185743E-06 0.000000000000000E+00 0.355691156040469E-03
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.214745783337898E-01 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.169314288346036E-05 0.000000000000000E+00 0.168698121829681E-02
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.783372206357044E-06 0.000000000000000E+00 0.628664387861163E-03
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.583931540240936E-06 0.000000000000000E+00 0.188756010107088E-02
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.635719820281023E-07 0.000000000000000E+00 0.862175514352166E-03
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.559031377154106E-01 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1728 STEPO   F00000000
 CREATE RESTART FILE ./srf00180000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00180000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00180000.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    1.6056 SECONDS******** Mbytes/s, TOTAL TIME=    2.1183(75.8%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00171200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1728 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001728 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001728
 MPI-TASK:   1 -    299876468 BYTES IN      6 RECORDS TRANSFERRED IN    0.0003 SECONDS999588.2 Mbytes/s, TOTAL TIME=    0.0134( 2.2%)
 NSTEP =  1728 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001728 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001728
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0059 SECONDS119898.2 Mbytes/s, TOTAL TIME=    0.2177( 2.7%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001728 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001728
 MPI-TASK:   1 -   4798023488 BYTES IN    125 RECORDS TRANSFERRED IN    0.0033 SECONDS******** Mbytes/s, TOTAL TIME=    0.1509( 2.2%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001728 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001728
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0001 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0029( 3.4%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001728 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001728
 MPI-TASK:   1 -    291296739 BYTES IN      4 RECORDS TRANSFERRED IN    0.0001 SECONDS******** Mbytes/s, TOTAL TIME=    0.0070( 1.4%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1728 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1728 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001728 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001728
 MPI-TASK:   1 -    299876468 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS999588.2 Mbytes/s, TOTAL TIME=    0.0079( 3.8%)
 08:04:45 STEP 1728 H= 432:00 +CPU= 26.982
 NSTEP =  1729 STEPO   0AAA00AAA
 08:05:05 STEP 1729 H= 432:15 +CPU= 20.454
ISTEP=    1730 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1730 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1730 STEPO   0AAA00AAA
 08:05:24 STEP 1730 H= 432:30 +CPU= 19.218
 NSTEP =  1731 STEPO   0AAA00AAA
 08:05:46 STEP 1731 H= 432:45 +CPU= 20.890
 NSTEP =  1732 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 08:04:24 FULLPOS-V DYNFPOS   1728     2.312    2.312    2.312    807:24    812:16 0.44038527185444E-05       2GB       0MB
 08:04:24 FULLPOS-T DYNFPOS   1728     0.445    0.445    0.445    807:24    812:16 0.44038527185444E-05       2GB       0MB
 08:04:25 0AAA00AAA STEPO     1728     0.441    0.441    0.443    807:25    812:17 0.44038527185444E-05       2GB       0MB
 08:04:45 0AAA00AAA STEPO     1729    20.137   20.137   20.140    807:45    812:37 0.44038527185444E-05       2GB       0MB
 08:05:05 0AAA00AAA STEPO     1730    20.454   20.454   20.456    808:05    812:57 0.44038527185444E-05       2GB       0MB
 08:05:24 0AAA00AAA STEPO     1731    19.218   19.218   19.219    808:24    813:17 0.44038527185444E-05       2GB       0MB
 08:05:46 0AAA00AAA STEPO     1732    20.891   20.891   21.116    808:45    813:38 0.44038527185444E-05       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1728",
 CTIME   = "00180000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1716, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/29/progress_file_12290072.xml

<running_values>
  <last_cpu_time>48527.910000</last_cpu_time>
  <upload_file_number>18</upload_file_number>
  <last_iter>1732</last_iter>
  <last_upload>1555200</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a1ch
(argv4) batchid: 1018
(argv5) wuid: 12290072
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/29
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/29/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/29/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9b79eb502bb62a6361d44a8918a74485 to: /var/lib/boinc-client/slots/29/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/29/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072.zip
ic_ancil_file: ic_ancil_12290072
ifsdata_file: ifsdata_12290072
climate_data_file: clim_data_12290072
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/29/ic_ancil_12290072.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/29/ic_ancil_12290072.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1ef9fd1a0cf78104a9388c17accb644e to: /var/lib/boinc-client/slots/29/ifsdata/ifsdata_12290072.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/29/ifsdata/ifsdata_12290072.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/29/159l_2/clim_data_12290072.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/29/159l_2/clim_data_12290072.zip
Checking for progress XML file: /var/lib/boinc-client/slots/29/progress_file_12290072.xml
Opened progress file ok : /var/lib/boinc-client/slots/29/progress_file_12290072.xml
-- Model is restarting --
Adjusting last_iter, 1732, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/29/progress_file_12290072.xml
last_cpu_time: 48527
upload_file_number: 18
last_iter: 1729
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0_r1487970834
The child process has been launched with process id: 2630092
Executing the command: /var/lib/boinc-client/slots/29/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:P5810:1:1:2630092:2630092] [20240623:124113:1719168073:0.001] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:41:56 STEP 1728 H= 432:00 +CPU= 37.444
  12:42:30 STEP 1729 H= 432:15 +CPU= 32.270
  12:43:08 STEP 1730 H= 432:30 +CPU= 36.933
  12:43:48 STEP 1731 H= 432:45 +CPU= 38.889
  12:44:31 STEP 1732 H= 433:00 +CPU= 41.399
  12:45:12 STEP 1733 H= 433:15 +CPU= 40.215
  12:45:42 STEP 1734 H= 433:30 +CPU= 29.531
  12:46:13 STEP 1735 H= 433:45 +CPU= 30.323
  12:46:54 STEP 1736 H= 434:00 +CPU= 39.363
  12:47:34 STEP 1737 H= 434:15 +CPU= 39.043
  12:48:13 STEP 1738 H= 434:30 +CPU= 38.431
  12:48:52 STEP 1739 H= 434:45 +CPU= 37.968
  12:49:32 STEP 1740 H= 435:00 +CPU= 38.798
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001740
  12:50:06 STEP 1741 H= 435:15 +CPU= 33.538
  12:50:46 STEP 1742 H= 435:30 +CPU= 38.754
  12:51:27 STEP 1743 H= 435:45 +CPU= 40.367
  12:52:08 STEP 1744 H= 436:00 +CPU= 39.696
  12:52:48 STEP 1745 H= 436:15 +CPU= 38.303
  12:53:29 STEP 1746 H= 436:30 +CPU= 40.356
  12:54:09 STEP 1747 H= 436:45 +CPU= 38.602
  12:54:46 STEP 1748 H= 437:00 +CPU= 36.833
  12:55:19 STEP 1749 H= 437:15 +CPU= 32.009
  12:55:57 STEP 1750 H= 437:30 +CPU= 37.363
  12:56:38 STEP 1751 H= 437:45 +CPU= 39.499
  12:57:27 STEP 1752 H= 438:00 +CPU= 47.594
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001752
  12:58:03 STEP 1753 H= 438:15 +CPU= 35.173
  12:58:42 STEP 1754 H= 438:30 +CPU= 37.840
  12:59:21 STEP 1755 H= 438:45 +CPU= 38.304
  13:00:01 STEP 1756 H= 439:00 +CPU= 38.491
  13:00:41 STEP 1757 H= 439:15 +CPU= 39.015
  13:01:20 STEP 1758 H= 439:30 +CPU= 38.450
  13:01:59 STEP 1759 H= 439:45 +CPU= 38.055
  13:02:38 STEP 1760 H= 440:00 +CPU= 37.680
  13:03:17 STEP 1761 H= 440:15 +CPU= 37.692
  13:03:51 STEP 1762 H= 440:30 +CPU= 34.001
  13:04:24 STEP 1763 H= 440:45 +CPU= 31.898
  13:05:12 STEP 1764 H= 441:00 +CPU= 46.732
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001764
  13:05:48 STEP 1765 H= 441:15 +CPU= 35.371
  13:06:28 STEP 1766 H= 441:30 +CPU= 38.789
  13:07:07 STEP 1767 H= 441:45 +CPU= 37.853
  13:07:46 STEP 1768 H= 442:00 +CPU= 38.102
  13:08:26 STEP 1769 H= 442:15 +CPU= 38.825
  13:09:07 STEP 1770 H= 442:30 +CPU= 39.369
  13:09:47 STEP 1771 H= 442:45 +CPU= 39.528
  13:10:28 STEP 1772 H= 443:00 +CPU= 39.471
  13:11:07 STEP 1773 H= 443:15 +CPU= 38.238
  13:11:49 STEP 1774 H= 443:30 +CPU= 40.479
  13:12:29 STEP 1775 H= 443:45 +CPU= 39.121
  13:13:23 STEP 1776 H= 444:00 +CPU= 53.226
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001776
  13:14:02 STEP 1777 H= 444:15 +CPU= 37.791
  13:14:44 STEP 1778 H= 444:30 +CPU= 40.148
  13:15:23 STEP 1779 H= 444:45 +CPU= 38.316
  13:16:02 STEP 1780 H= 445:00 +CPU= 37.459
  13:16:41 STEP 1781 H= 445:15 +CPU= 38.024
  13:17:16 STEP 1782 H= 445:30 +CPU= 34.984
  13:17:51 STEP 1783 H= 445:45 +CPU= 33.404
  13:18:30 STEP 1784 H= 446:00 +CPU= 38.175
  13:19:09 STEP 1785 H= 446:15 +CPU= 38.557
  13:19:49 STEP 1786 H= 446:30 +CPU= 38.988
  13:20:30 STEP 1787 H= 446:45 +CPU= 39.694
  13:21:19 STEP 1788 H= 447:00 +CPU= 47.958
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001788
  13:21:58 STEP 1789 H= 447:15 +CPU= 37.332
  13:22:37 STEP 1790 H= 447:30 +CPU= 38.573
  13:23:16 STEP 1791 H= 447:45 +CPU= 38.234
  13:23:55 STEP 1792 H= 448:00 +CPU= 37.273
  13:24:37 STEP 1793 H= 448:15 +CPU= 40.694
  13:25:16 STEP 1794 H= 448:30 +CPU= 38.002
  13:25:52 STEP 1795 H= 448:45 +CPU= 35.284
  13:26:31 STEP 1796 H= 449:00 +CPU= 38.245
  13:27:09 STEP 1797 H= 449:15 +CPU= 36.700
  13:27:49 STEP 1798 H= 449:30 +CPU= 38.285
  13:28:28 STEP 1799 H= 449:45 +CPU= 38.683
  13:29:16 STEP 1800 H= 450:00 +CPU= 46.863
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001800
  13:29:53 STEP 1801 H= 450:15 +CPU= 35.891
  13:30:33 STEP 1802 H= 450:30 +CPU= 38.918
  13:31:11 STEP 1803 H= 450:45 +CPU= 37.198
  13:31:45 STEP 1804 H= 451:00 +CPU= 32.842
  13:32:20 STEP 1805 H= 451:15 +CPU= 34.553
  13:33:01 STEP 1806 H= 451:30 +CPU= 39.374
  13:33:41 STEP 1807 H= 451:45 +CPU= 39.363
  13:34:23 STEP 1808 H= 452:00 +CPU= 40.804
  13:35:00 STEP 1809 H= 452:15 +CPU= 36.084
  13:35:39 STEP 1810 H= 452:30 +CPU= 37.948
  13:36:18 STEP 1811 H= 452:45 +CPU= 38.521
  13:37:09 STEP 1812 H= 453:00 +CPU= 49.177
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001812
  13:37:50 STEP 1813 H= 453:15 +CPU= 40.015
  13:38:27 STEP 1814 H= 453:30 +CPU= 35.508
  13:39:04 STEP 1815 H= 453:45 +CPU= 36.522
  13:39:43 STEP 1816 H= 454:00 +CPU= 37.907
  13:40:23 STEP 1817 H= 454:15 +CPU= 38.886
  13:41:01 STEP 1818 H= 454:30 +CPU= 36.742
  13:41:40 STEP 1819 H= 454:45 +CPU= 38.074
  13:42:22 STEP 1820 H= 455:00 +CPU= 40.681
  13:43:03 STEP 1821 H= 455:15 +CPU= 39.829
  13:43:43 STEP 1822 H= 455:30 +CPU= 38.485
  13:44:22 STEP 1823 H= 455:45 +CPU= 37.531
  13:45:16 STEP 1824 H= 456:00 +CPU= 52.849
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0_r1487970834_18.zip
Uploading the intermediate file: upload_file_18.zip
  13:45:54 STEP 1825 H= 456:15 +CPU= 36.662
  13:46:36 STEP 1826 H= 456:30 +CPU= 40.500
  13:47:15 STEP 1827 H= 456:45 +CPU= 37.382
  13:47:54 STEP 1828 H= 457:00 +CPU= 38.251
  13:48:31 STEP 1829 H= 457:15 +CPU= 35.545
  13:49:09 STEP 1830 H= 457:30 +CPU= 37.321
  13:49:47 STEP 1831 H= 457:45 +CPU= 36.504
  13:50:27 STEP 1832 H= 458:00 +CPU= 38.269
  13:51:04 STEP 1833 H= 458:15 +CPU= 36.222
  13:51:43 STEP 1834 H= 458:30 +CPU= 37.506
  13:52:21 STEP 1835 H= 458:45 +CPU= 37.818
  13:53:10 STEP 1836 H= 459:00 +CPU= 47.287
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001836
  13:53:50 STEP 1837 H= 459:15 +CPU= 38.505
  13:54:28 STEP 1838 H= 459:30 +CPU= 37.072
  13:55:09 STEP 1839 H= 459:45 +CPU= 39.041
  13:55:48 STEP 1840 H= 460:00 +CPU= 38.238
  13:56:25 STEP 1841 H= 460:15 +CPU= 36.073
  13:57:04 STEP 1842 H= 460:30 +CPU= 37.493
  13:57:43 STEP 1843 H= 460:45 +CPU= 38.094
  13:58:24 STEP 1844 H= 461:00 +CPU= 40.157
  13:59:03 STEP 1845 H= 461:15 +CPU= 37.891
  13:59:43 STEP 1846 H= 461:30 +CPU= 38.517
  14:00:22 STEP 1847 H= 461:45 +CPU= 38.172
  14:01:11 STEP 1848 H= 462:00 +CPU= 47.274
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001848
  14:01:49 STEP 1849 H= 462:15 +CPU= 36.901
  14:02:28 STEP 1850 H= 462:30 +CPU= 38.421
  14:03:07 STEP 1851 H= 462:45 +CPU= 38.063
  14:03:47 STEP 1852 H= 463:00 +CPU= 38.788
  14:04:27 STEP 1853 H= 463:15 +CPU= 38.769
  14:05:07 STEP 1854 H= 463:30 +CPU= 39.237
  14:05:47 STEP 1855 H= 463:45 +CPU= 38.934
  14:06:28 STEP 1856 H= 464:00 +CPU= 40.234
  14:07:07 STEP 1857 H= 464:15 +CPU= 37.041
  14:07:45 STEP 1858 H= 464:30 +CPU= 37.332
  14:08:24 STEP 1859 H= 464:45 +CPU= 37.817
  14:09:14 STEP 1860 H= 465:00 +CPU= 48.302
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001860
  14:09:52 STEP 1861 H= 465:15 +CPU= 37.054
  14:10:31 STEP 1862 H= 465:30 +CPU= 38.482
  14:11:12 STEP 1863 H= 465:45 +CPU= 39.316
  14:11:49 STEP 1864 H= 466:00 +CPU= 35.869
  14:12:28 STEP 1865 H= 466:15 +CPU= 38.180
  14:13:09 STEP 1866 H= 466:30 +CPU= 39.433
  14:13:48 STEP 1867 H= 466:45 +CPU= 37.662
  14:14:28 STEP 1868 H= 467:00 +CPU= 38.959
  14:15:08 STEP 1869 H= 467:15 +CPU= 39.590
  14:15:50 STEP 1870 H= 467:30 +CPU= 40.132
  14:16:30 STEP 1871 H= 467:45 +CPU= 39.093
  14:17:25 STEP 1872 H= 468:00 +CPU= 53.766
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001872
  14:18:04 STEP 1873 H= 468:15 +CPU= 37.583
  14:18:42 STEP 1874 H= 468:30 +CPU= 36.713
  14:19:22 STEP 1875 H= 468:45 +CPU= 38.871
  14:20:02 STEP 1876 H= 469:00 +CPU= 39.025
  14:20:41 STEP 1877 H= 469:15 +CPU= 37.867
  14:21:19 STEP 1878 H= 469:30 +CPU= 36.702
  14:21:57 STEP 1879 H= 469:45 +CPU= 37.193
  14:22:37 STEP 1880 H= 470:00 +CPU= 38.724
  14:23:16 STEP 1881 H= 470:15 +CPU= 38.509
  14:23:57 STEP 1882 H= 470:30 +CPU= 39.395
  14:24:37 STEP 1883 H= 470:45 +CPU= 38.801
  14:25:25 STEP 1884 H= 471:00 +CPU= 46.671
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001884
  14:26:05 STEP 1885 H= 471:15 +CPU= 38.865
  14:26:45 STEP 1886 H= 471:30 +CPU= 38.673
  14:27:24 STEP 1887 H= 471:45 +CPU= 37.974
  14:28:04 STEP 1888 H= 472:00 +CPU= 39.585
  14:28:45 STEP 1889 H= 472:15 +CPU= 39.767
  14:29:25 STEP 1890 H= 472:30 +CPU= 38.266
  14:30:05 STEP 1891 H= 472:45 +CPU= 39.592
  14:30:46 STEP 1892 H= 473:00 +CPU= 39.415
  14:31:24 STEP 1893 H= 473:15 +CPU= 37.244
  14:32:03 STEP 1894 H= 473:30 +CPU= 37.773
  14:32:42 STEP 1895 H= 473:45 +CPU= 38.365
  14:33:33 STEP 1896 H= 474:00 +CPU= 49.785
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001896
  14:34:15 STEP 1897 H= 474:15 +CPU= 40.198
  14:34:54 STEP 1898 H= 474:30 +CPU= 38.648
  14:35:32 STEP 1899 H= 474:45 +CPU= 36.016
  14:36:14 STEP 1900 H= 475:00 +CPU= 40.881
  14:36:55 STEP 1901 H= 475:15 +CPU= 39.761
  14:37:33 STEP 1902 H= 475:30 +CPU= 37.151
  14:38:13 STEP 1903 H= 475:45 +CPU= 39.484
  14:38:54 STEP 1904 H= 476:00 +CPU= 39.371
  14:39:34 STEP 1905 H= 476:15 +CPU= 38.899
  14:40:13 STEP 1906 H= 476:30 +CPU= 38.302
  14:40:53 STEP 1907 H= 476:45 +CPU= 38.917
  14:41:41 STEP 1908 H= 477:00 +CPU= 46.975
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001908
  14:42:22 STEP 1909 H= 477:15 +CPU= 39.259
  14:43:01 STEP 1910 H= 477:30 +CPU= 38.083
  14:43:41 STEP 1911 H= 477:45 +CPU= 39.028
  14:44:21 STEP 1912 H= 478:00 +CPU= 38.642
  14:44:59 STEP 1913 H= 478:15 +CPU= 36.924
  14:45:36 STEP 1914 H= 478:30 +CPU= 36.554
  14:46:14 STEP 1915 H= 478:45 +CPU= 36.137
  14:46:52 STEP 1916 H= 479:00 +CPU= 37.195
  14:47:31 STEP 1917 H= 479:15 +CPU= 38.735
  14:48:10 STEP 1918 H= 479:30 +CPU= 37.749
  14:48:51 STEP 1919 H= 479:45 +CPU= 39.891
  14:49:42 STEP 1920 H= 480:00 +CPU= 50.116
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.4556183742E-21      I       0.3115027801E-05      I
ICloud fractionI      0.0000000000E+00      I       0.3314117925E-01      I
I  Relat. Hum. I      0.1322077343E+00      I       0.3114066574E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.6887858686E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    300671636 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS429530.9 Mbytes/s, TOTAL TIME=    0.0156( 4.5%)
 14:49:42 STEP 1920 H= 480:00 +CPU= 50.116
 END CNT3
 NSTEP =  1920 CNT0   000000000  36.756  36.756  37.509   0.980               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         5720/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    103    0       46      1044068     5720 ************    15.8     2.3    13.4     3.2    28.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 7709404.6 7709404.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 7704366.5 7704366.5     99.93      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209    5505.5    5505.5     14.93      0.00
   9 SPCM     - SPECTRAL COMP.                     192    1089.2    1089.2      2.71      0.00
  10 SCAN2M - PHYSICS                              193   20505.4   20505.4     51.33      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16    1116.7    1116.7      0.23      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  9      62.2      62.2      0.01      0.00
  17 GRIDFPOS IN CNT4                               16      35.9      35.9      0.01      0.00
  27 RESTART FILES - WRITING                         4    4466.6    4466.6      0.23      0.00
  28 RESTART FILES - READING                         1    4203.2    4203.2      0.05      0.00
  29 SU4FPOS IN CNT4                                16       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                16    8411.0    8411.0      1.75      0.00
  31 POSDDH IN STEPO                                 1       1.4       1.4      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193    8044.4    8044.4     20.14      0.00
  39 SU0YOMB                                         1     557.9     557.9      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      43.8      43.8      0.11      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192     192.5     192.5      0.48      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192      70.2      70.2      0.17      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  9       0.3       0.3      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      420     628.6     628.6      3.42      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       400     359.3     359.3      1.86      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        400     132.2     132.2      0.69      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       420     205.0     205.0      1.12      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      63.7      63.7      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            420      69.8      69.8      0.38      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            400      35.0      35.0      0.18      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            420      97.2      97.2      0.53      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            400      30.0      30.0      0.16      0.00
 400 GSTATS                                      23311       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 21635       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     7709.4 CPU TIME    7498.9 VECTOR TIME     7498.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      224.3
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc-client/slots/29/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/slots/29/NODE.001_01
Adding to the zip: /var/lib/boinc-client/slots/29/ifs.stat
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12290072/ICMUAh7zg+001896
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0_r1487970834_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
14:53:19 (2629933): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 Jun 2024 21:49:23 1551031 22443661 oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0 1,727,100 56,026 0.0324
23 Jun 2024 09:49:54 1551031 22443661 oifs_43r3_bl_a1ch_2016092300_20_1018_12290072_0 864,000 32,049 0.0371


©2024 climateprediction.net