climateprediction.net home page
Task 22446283

Task 22446283

Name oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0
Workunit 12292674
Created 13 Jun 2024, 2:19:08 UTC
Sent 13 Jun 2024, 12:54:29 UTC
Report deadline 12 Aug 2024, 12:54:29 UTC
Received 14 Jun 2024, 1:36:18 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1504810
Run time 10 hours 39 min 12 sec
CPU time 10 hours 37 min 16 sec
Validate state Valid
Credit 2,638.30
Device peak FLOPS 3.50 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,423.61 MB
Peak swap size 5,981.07 MB
Peak disk usage 1,372.13 MB
Stderr
<core_client_version>7.22.1</core_client_version>
<![CDATA[
<stderr_txt>
at (nil))
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:562483:562483] [20240614:002117:1718317277:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:21:40 STEP 1536 H= 384:00 +CPU= 20.462
  00:21:59 STEP 1537 H= 384:15 +CPU= 19.764
  00:22:19 STEP 1538 H= 384:30 +CPU= 19.580
  00:22:38 STEP 1539 H= 384:45 +CPU= 18.965
  00:22:58 STEP 1540 H= 385:00 +CPU= 19.612
  00:23:17 STEP 1541 H= 385:15 +CPU= 19.361
  00:23:37 STEP 1542 H= 385:30 +CPU= 19.795
  00:23:57 STEP 1543 H= 385:45 +CPU= 19.388
  00:24:17 STEP 1544 H= 386:00 +CPU= 19.945
  00:24:36 STEP 1545 H= 386:15 +CPU= 19.006
  00:24:55 STEP 1546 H= 386:30 +CPU= 19.316
  00:25:14 STEP 1547 H= 386:45 +CPU= 19.133
  00:25:39 STEP 1548 H= 387:00 +CPU= 25.112
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001548
  00:25:59 STEP 1549 H= 387:15 +CPU= 19.297
  00:26:18 STEP 1550 H= 387:30 +CPU= 19.381
  00:26:37 STEP 1551 H= 387:45 +CPU= 19.320
  00:26:57 STEP 1552 H= 388:00 +CPU= 19.754
  00:27:16 STEP 1553 H= 388:15 +CPU= 19.261
  00:27:36 STEP 1554 H= 388:30 +CPU= 19.395
  00:27:55 STEP 1555 H= 388:45 +CPU= 19.566
  00:28:15 STEP 1556 H= 389:00 +CPU= 19.723
  00:28:34 STEP 1557 H= 389:15 +CPU= 18.794
  00:28:53 STEP 1558 H= 389:30 +CPU= 19.216
  00:29:13 STEP 1559 H= 389:45 +CPU= 19.638
  00:29:39 STEP 1560 H= 390:00 +CPU= 25.918
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001560
  00:29:58 STEP 1561 H= 390:15 +CPU= 19.259
  00:30:17 STEP 1562 H= 390:30 +CPU= 19.016
  00:30:37 STEP 1563 H= 390:45 +CPU= 19.272
  00:30:57 STEP 1564 H= 391:00 +CPU= 20.267
  00:31:16 STEP 1565 H= 391:15 +CPU= 19.056
  00:31:35 STEP 1566 H= 391:30 +CPU= 19.352
  00:31:55 STEP 1567 H= 391:45 +CPU= 19.680
  00:32:15 STEP 1568 H= 392:00 +CPU= 20.248
  00:32:35 STEP 1569 H= 392:15 +CPU= 19.881
  00:32:55 STEP 1570 H= 392:30 +CPU= 19.609
  00:33:14 STEP 1571 H= 392:45 +CPU= 18.983
  00:33:39 STEP 1572 H= 393:00 +CPU= 24.699
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001572
  00:33:58 STEP 1573 H= 393:15 +CPU= 19.503
  00:34:18 STEP 1574 H= 393:30 +CPU= 19.337
  00:34:37 STEP 1575 H= 393:45 +CPU= 19.161
  00:34:57 STEP 1576 H= 394:00 +CPU= 19.646
  00:35:16 STEP 1577 H= 394:15 +CPU= 19.206
  00:35:35 STEP 1578 H= 394:30 +CPU= 19.423
  00:35:55 STEP 1579 H= 394:45 +CPU= 19.229
  00:36:14 STEP 1580 H= 395:00 +CPU= 19.747
  00:36:34 STEP 1581 H= 395:15 +CPU= 19.237
  00:36:53 STEP 1582 H= 395:30 +CPU= 19.685
  00:37:13 STEP 1583 H= 395:45 +CPU= 19.486
  00:37:40 STEP 1584 H= 396:00 +CPU= 27.115
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001584
  00:37:59 STEP 1585 H= 396:15 +CPU= 18.902
  00:38:18 STEP 1586 H= 396:30 +CPU= 19.125
  00:38:38 STEP 1587 H= 396:45 +CPU= 19.919
  00:38:59 STEP 1588 H= 397:00 +CPU= 20.545
  00:39:18 STEP 1589 H= 397:15 +CPU= 19.610
  00:39:38 STEP 1590 H= 397:30 +CPU= 19.710
  00:39:57 STEP 1591 H= 397:45 +CPU= 18.923
  00:40:17 STEP 1592 H= 398:00 +CPU= 19.808
  00:40:36 STEP 1593 H= 398:15 +CPU= 19.093
  00:40:55 STEP 1594 H= 398:30 +CPU= 19.433
  00:41:15 STEP 1595 H= 398:45 +CPU= 19.950
  00:41:41 STEP 1596 H= 399:00 +CPU= 25.347
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001596
  00:42:00 STEP 1597 H= 399:15 +CPU= 18.933
  00:42:19 STEP 1598 H= 399:30 +CPU= 19.370
  00:42:39 STEP 1599 H= 399:45 +CPU= 19.358
  00:42:59 STEP 1600 H= 400:00 +CPU= 20.283
  00:43:18 STEP 1601 H= 400:15 +CPU= 19.300
  00:43:38 STEP 1602 H= 400:30 +CPU= 19.399
  00:43:57 STEP 1603 H= 400:45 +CPU= 19.036
  00:44:17 STEP 1604 H= 401:00 +CPU= 19.796
  00:44:36 STEP 1605 H= 401:15 +CPU= 19.068
  00:44:55 STEP 1606 H= 401:30 +CPU= 19.382
  00:45:15 STEP 1607 H= 401:45 +CPU= 19.641
  00:45:41 STEP 1608 H= 402:00 +CPU= 25.662
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001608
  00:46:00 STEP 1609 H= 402:15 +CPU= 19.343
  00:46:20 STEP 1610 H= 402:30 +CPU= 19.685
  00:46:39 STEP 1611 H= 402:45 +CPU= 19.674
  00:47:00 STEP 1612 H= 403:00 +CPU= 20.363
  00:47:19 STEP 1613 H= 403:15 +CPU= 19.430
  00:47:39 STEP 1614 H= 403:30 +CPU= 19.656
  00:47:59 STEP 1615 H= 403:45 +CPU= 19.647
  00:48:19 STEP 1616 H= 404:00 +CPU= 20.494
  00:48:39 STEP 1617 H= 404:15 +CPU= 19.448
  00:48:58 STEP 1618 H= 404:30 +CPU= 19.391
  00:49:17 STEP 1619 H= 404:45 +CPU= 19.229
  00:49:42 STEP 1620 H= 405:00 +CPU= 24.700
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001620
  00:50:02 STEP 1621 H= 405:15 +CPU= 19.546
  00:50:21 STEP 1622 H= 405:30 +CPU= 19.408
  00:50:40 STEP 1623 H= 405:45 +CPU= 19.393
  00:51:01 STEP 1624 H= 406:00 +CPU= 20.183
  00:51:20 STEP 1625 H= 406:15 +CPU= 19.474
  00:51:40 STEP 1626 H= 406:30 +CPU= 19.300
  00:51:59 STEP 1627 H= 406:45 +CPU= 19.529
  00:52:19 STEP 1628 H= 407:00 +CPU= 20.317
  00:52:39 STEP 1629 H= 407:15 +CPU= 18.991
  00:52:58 STEP 1630 H= 407:30 +CPU= 19.429
  00:53:17 STEP 1631 H= 407:45 +CPU= 19.077
  00:53:45 STEP 1632 H= 408:00 +CPU= 27.651
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001632
  00:54:05 STEP 1633 H= 408:15 +CPU= 19.677
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001632
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0_r1419436944_16.zip
Uploading the intermediate file: upload_file_16.zip
  00:54:24 STEP 1634 H= 408:30 +CPU= 19.901
  00:54:44 STEP 1635 H= 408:45 +CPU= 19.060
  00:55:04 STEP 1636 H= 409:00 +CPU= 20.572
  00:55:24 STEP 1637 H= 409:15 +CPU= 19.587
  00:55:43 STEP 1638 H= 409:30 +CPU= 18.909
  00:56:02 STEP 1639 H= 409:45 +CPU= 18.916
  00:56:22 STEP 1640 H= 410:00 +CPU= 20.251
  00:56:42 STEP 1641 H= 410:15 +CPU= 19.578
  00:57:01 STEP 1642 H= 410:30 +CPU= 19.436
  00:57:20 STEP 1643 H= 410:45 +CPU= 19.129
  00:57:45 STEP 1644 H= 411:00 +CPU= 24.993
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001644
  00:58:05 STEP 1645 H= 411:15 +CPU= 19.432
  00:58:24 STEP 1646 H= 411:30 +CPU= 19.243
  00:58:43 STEP 1647 H= 411:45 +CPU= 18.986
  00:59:03 STEP 1648 H= 412:00 +CPU= 20.295
  00:59:23 STEP 1649 H= 412:15 +CPU= 19.818
  00:59:43 STEP 1650 H= 412:30 +CPU= 20.219
  01:00:03 STEP 1651 H= 412:45 +CPU= 19.913
  01:00:23 STEP 1652 H= 413:00 +CPU= 19.908
  01:00:43 STEP 1653 H= 413:15 +CPU= 19.301
  01:01:02 STEP 1654 H= 413:30 +CPU= 19.642
  01:01:21 STEP 1655 H= 413:45 +CPU= 18.950
  01:01:47 STEP 1656 H= 414:00 +CPU= 25.197
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001656
  01:02:06 STEP 1657 H= 414:15 +CPU= 19.374
  01:02:26 STEP 1658 H= 414:30 +CPU= 19.635
  01:02:46 STEP 1659 H= 414:45 +CPU= 20.136
  01:03:06 STEP 1660 H= 415:00 +CPU= 20.078
  01:03:25 STEP 1661 H= 415:15 +CPU= 19.432
  01:03:45 STEP 1662 H= 415:30 +CPU= 19.209
  01:04:04 STEP 1663 H= 415:45 +CPU= 19.528
  01:04:25 STEP 1664 H= 416:00 +CPU= 20.428
  01:04:44 STEP 1665 H= 416:15 +CPU= 19.628
  01:05:04 STEP 1666 H= 416:30 +CPU= 19.335
  01:05:23 STEP 1667 H= 416:45 +CPU= 19.200
  01:05:48 STEP 1668 H= 417:00 +CPU= 24.993
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001668
  01:06:07 STEP 1669 H= 417:15 +CPU= 19.307
  01:06:27 STEP 1670 H= 417:30 +CPU= 19.553
  01:06:46 STEP 1671 H= 417:45 +CPU= 19.166
  01:07:06 STEP 1672 H= 418:00 +CPU= 20.142
  01:07:26 STEP 1673 H= 418:15 +CPU= 19.727
  01:07:46 STEP 1674 H= 418:30 +CPU= 19.626
  01:08:06 STEP 1675 H= 418:45 +CPU= 19.923
  01:08:27 STEP 1676 H= 419:00 +CPU= 20.874
  01:08:46 STEP 1677 H= 419:15 +CPU= 19.496
  01:09:06 STEP 1678 H= 419:30 +CPU= 19.418
  01:09:25 STEP 1679 H= 419:45 +CPU= 19.173
  01:09:53 STEP 1680 H= 420:00 +CPU= 27.458
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001680
  01:10:12 STEP 1681 H= 420:15 +CPU= 19.440
  01:10:32 STEP 1682 H= 420:30 +CPU= 19.982
  01:10:52 STEP 1683 H= 420:45 +CPU= 19.570
  01:11:12 STEP 1684 H= 421:00 +CPU= 19.896
  01:11:31 STEP 1685 H= 421:15 +CPU= 19.156
  01:11:50 STEP 1686 H= 421:30 +CPU= 19.675
  01:12:10 STEP 1687 H= 421:45 +CPU= 19.197
  01:12:30 STEP 1688 H= 422:00 +CPU= 19.974
  01:12:50 STEP 1689 H= 422:15 +CPU= 19.763
  01:13:09 STEP 1690 H= 422:30 +CPU= 19.896
  01:13:29 STEP 1691 H= 422:45 +CPU= 19.742
  01:13:54 STEP 1692 H= 423:00 +CPU= 24.913
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001692
  01:14:13 STEP 1693 H= 423:15 +CPU= 19.256
  01:14:33 STEP 1694 H= 423:30 +CPU= 19.203
  01:14:52 STEP 1695 H= 423:45 +CPU= 19.148
  01:15:12 STEP 1696 H= 424:00 +CPU= 19.776
  01:15:31 STEP 1697 H= 424:15 +CPU= 19.376
  01:15:51 STEP 1698 H= 424:30 +CPU= 19.606
  01:16:11 STEP 1699 H= 424:45 +CPU= 20.032
  01:16:31 STEP 1700 H= 425:00 +CPU= 20.085
  01:16:50 STEP 1701 H= 425:15 +CPU= 18.992
  01:17:10 STEP 1702 H= 425:30 +CPU= 19.522
  01:17:29 STEP 1703 H= 425:45 +CPU= 19.227
  01:17:54 STEP 1704 H= 426:00 +CPU= 25.542
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001704
  01:18:14 STEP 1705 H= 426:15 +CPU= 19.450
  01:18:33 STEP 1706 H= 426:30 +CPU= 19.509
  01:18:53 STEP 1707 H= 426:45 +CPU= 19.814
  01:19:14 STEP 1708 H= 427:00 +CPU= 20.363
  01:19:33 STEP 1709 H= 427:15 +CPU= 19.412
  01:19:52 STEP 1710 H= 427:30 +CPU= 19.174
  01:20:12 STEP 1711 H= 427:45 +CPU= 19.417
  01:20:32 STEP 1712 H= 428:00 +CPU= 20.616
  01:20:52 STEP 1713 H= 428:15 +CPU= 19.819
  01:21:12 STEP 1714 H= 428:30 +CPU= 19.518
  01:21:32 STEP 1715 H= 428:45 +CPU= 19.820
  01:21:58 STEP 1716 H= 429:00 +CPU= 26.026
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001716
  01:22:17 STEP 1717 H= 429:15 +CPU= 19.489
  01:22:37 STEP 1718 H= 429:30 +CPU= 19.352
  01:22:56 STEP 1719 H= 429:45 +CPU= 19.053
  01:23:16 STEP 1720 H= 430:00 +CPU= 20.557
  01:23:36 STEP 1721 H= 430:15 +CPU= 19.564
  01:23:56 STEP 1722 H= 430:30 +CPU= 19.601
  01:24:15 STEP 1723 H= 430:45 +CPU= 19.418
  01:24:35 STEP 1724 H= 431:00 +CPU= 20.017
  01:24:54 STEP 1725 H= 431:15 +CPU= 19.038
  01:25:14 STEP 1726 H= 431:30 +CPU= 19.782
  01:25:34 STEP 1727 H= 431:45 +CPU= 19.676
  01:26:02 STEP 1728 H= 432:00 +CPU= 27.827
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001728
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3cr
(argv4) batchid: 1018
(argv5) wuid: 12292674
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_042aa3a05d002afae763a06085c2491d to: /var/lib/boinc/slots/1/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674.zip
ic_ancil_file: ic_ancil_12292674
ifsdata_file: ifsdata_12292674
climate_data_file: clim_data_12292674
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/1/ic_ancil_12292674.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12292674.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_088e67a42cd2dfccbf299ef0e824d68c to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12292674.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12292674.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12292674.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12292674.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12292674.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12292674.xml
-- Model is restarting --
Adjusting last_iter, 1729, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12292674.xml
last_cpu_time: 34321
upload_file_number: 17
last_iter: 1729
last_upload: 1469700
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0_r1419436944
The child process has been launched with process id: 565567
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:565567:565567] [20240614:012717:1718321237:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001728
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0_r1419436944_17.zip
  01:27:41 STEP 1728 H= 432:00 +CPU= 20.759
Uploading the intermediate file: upload_file_17.zip
  01:28:00 STEP 1729 H= 432:15 +CPU= 19.616
  01:28:20 STEP 1730 H= 432:30 +CPU= 19.337
  01:28:39 STEP 1731 H= 432:45 +CPU= 19.184
  01:28:59 STEP 1732 H= 433:00 +CPU= 19.727
  01:29:18 STEP 1733 H= 433:15 +CPU= 19.180
  01:29:37 STEP 1734 H= 433:30 +CPU= 19.589
  01:30:02 STEP 1735 H= 433:45 +CPU= 23.644
  01:30:22 STEP 1736 H= 434:00 +CPU= 20.443
  01:30:42 STEP 1737 H= 434:15 +CPU= 19.318
  01:31:01 STEP 1738 H= 434:30 +CPU= 19.336
  01:31:20 STEP 1739 H= 434:45 +CPU= 19.355
  01:31:46 STEP 1740 H= 435:00 +CPU= 25.844
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001740
  01:32:06 STEP 1741 H= 435:15 +CPU= 19.930
  01:32:26 STEP 1742 H= 435:30 +CPU= 19.492
  01:32:45 STEP 1743 H= 435:45 +CPU= 19.718
  01:33:06 STEP 1744 H= 436:00 +CPU= 20.050
  01:33:26 STEP 1745 H= 436:15 +CPU= 20.020
  01:33:46 STEP 1746 H= 436:30 +CPU= 19.901
  01:34:05 STEP 1747 H= 436:45 +CPU= 19.817
  01:34:26 STEP 1748 H= 437:00 +CPU= 20.384
  01:34:46 STEP 1749 H= 437:15 +CPU= 19.977
  01:35:05 STEP 1750 H= 437:30 +CPU= 19.464
  01:35:25 STEP 1751 H= 437:45 +CPU= 19.380
  01:35:50 STEP 1752 H= 438:00 +CPU= 25.639
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001752
  01:36:10 STEP 1753 H= 438:15 +CPU= 19.361
  01:36:29 STEP 1754 H= 438:30 +CPU= 19.329
  01:36:48 STEP 1755 H= 438:45 +CPU= 19.225
  01:37:09 STEP 1756 H= 439:00 +CPU= 20.434
  01:37:29 STEP 1757 H= 439:15 +CPU= 19.673
  01:37:48 STEP 1758 H= 439:30 +CPU= 19.427
  01:38:08 STEP 1759 H= 439:45 +CPU= 19.620
  01:38:28 STEP 1760 H= 440:00 +CPU= 20.217
  01:38:47 STEP 1761 H= 440:15 +CPU= 19.262
  01:39:07 STEP 1762 H= 440:30 +CPU= 19.373
  01:39:26 STEP 1763 H= 440:45 +CPU= 19.337
  01:39:52 STEP 1764 H= 441:00 +CPU= 25.800
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001764
  01:40:12 STEP 1765 H= 441:15 +CPU= 19.758
  01:40:32 STEP 1766 H= 441:30 +CPU= 19.994
  01:40:52 STEP 1767 H= 441:45 +CPU= 19.733
  01:41:12 STEP 1768 H= 442:00 +CPU= 20.537
  01:41:32 STEP 1769 H= 442:15 +CPU= 19.313
  01:41:51 STEP 1770 H= 442:30 +CPU= 19.194
  01:42:10 STEP 1771 H= 442:45 +CPU= 19.269
  01:42:31 STEP 1772 H= 443:00 +CPU= 20.517
  01:42:50 STEP 1773 H= 443:15 +CPU= 19.484
  01:43:09 STEP 1774 H= 443:30 +CPU= 19.239
  01:43:29 STEP 1775 H= 443:45 +CPU= 19.148
  01:43:56 STEP 1776 H= 444:00 +CPU= 27.342
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001776
  01:44:16 STEP 1777 H= 444:15 +CPU= 19.710
  01:44:35 STEP 1778 H= 444:30 +CPU= 19.426
  01:44:55 STEP 1779 H= 444:45 +CPU= 19.317
  01:45:15 STEP 1780 H= 445:00 +CPU= 20.455
  01:45:35 STEP 1781 H= 445:15 +CPU= 19.541
  01:45:54 STEP 1782 H= 445:30 +CPU= 19.212
  01:46:13 STEP 1783 H= 445:45 +CPU= 19.174
  01:46:33 STEP 1784 H= 446:00 +CPU= 20.147
  01:46:53 STEP 1785 H= 446:15 +CPU= 19.568
  01:47:12 STEP 1786 H= 446:30 +CPU= 19.652
  01:47:32 STEP 1787 H= 446:45 +CPU= 19.054
  01:47:57 STEP 1788 H= 447:00 +CPU= 25.306
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001788
  01:48:16 STEP 1789 H= 447:15 +CPU= 19.187
  01:48:36 STEP 1790 H= 447:30 +CPU= 19.453
  01:48:56 STEP 1791 H= 447:45 +CPU= 19.985
  01:49:16 STEP 1792 H= 448:00 +CPU= 20.270
  01:49:35 STEP 1793 H= 448:15 +CPU= 19.477
  01:49:55 STEP 1794 H= 448:30 +CPU= 19.770
  01:50:15 STEP 1795 H= 448:45 +CPU= 19.321
  01:50:35 STEP 1796 H= 449:00 +CPU= 20.255
  01:50:55 STEP 1797 H= 449:15 +CPU= 20.199
  01:51:15 STEP 1798 H= 449:30 +CPU= 19.852
  01:51:35 STEP 1799 H= 449:45 +CPU= 20.269
  01:52:01 STEP 1800 H= 450:00 +CPU= 25.544
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001800
  01:52:21 STEP 1801 H= 450:15 +CPU= 19.485
  01:52:40 STEP 1802 H= 450:30 +CPU= 19.662
  01:53:00 STEP 1803 H= 450:45 +CPU= 19.436
  01:53:20 STEP 1804 H= 451:00 +CPU= 20.026
  01:53:39 STEP 1805 H= 451:15 +CPU= 18.929
  01:53:58 STEP 1806 H= 451:30 +CPU= 19.258
  01:54:17 STEP 1807 H= 451:45 +CPU= 19.265
  01:54:38 STEP 1808 H= 452:00 +CPU= 20.734
  01:54:58 STEP 1809 H= 452:15 +CPU= 19.713
  01:55:17 STEP 1810 H= 452:30 +CPU= 19.416
  01:55:37 STEP 1811 H= 452:45 +CPU= 19.459
  01:56:03 STEP 1812 H= 453:00 +CPU= 26.115
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001812
  01:56:23 STEP 1813 H= 453:15 +CPU= 19.955
  01:56:43 STEP 1814 H= 453:30 +CPU= 19.817
  01:57:03 STEP 1815 H= 453:45 +CPU= 19.785
  01:57:23 STEP 1816 H= 454:00 +CPU= 20.408
  01:57:43 STEP 1817 H= 454:15 +CPU= 19.862
  01:58:03 STEP 1818 H= 454:30 +CPU= 19.865
  01:58:23 STEP 1819 H= 454:45 +CPU= 19.849
  01:58:43 STEP 1820 H= 455:00 +CPU= 20.491
  01:59:03 STEP 1821 H= 455:15 +CPU= 19.859
  01:59:23 STEP 1822 H= 455:30 +CPU= 19.680
  01:59:42 STEP 1823 H= 455:45 +CPU= 19.592
  02:00:10 STEP 1824 H= 456:00 +CPU= 27.468
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001824
  02:00:29 STEP 1825 H= 456:15 +CPU= 19.420
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001824
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0_r1419436944_18.zip
  02:00:49 STEP 1826 H= 456:30 +CPU= 19.284
Uploading the intermediate file: upload_file_18.zip
  02:01:08 STEP 1827 H= 456:45 +CPU= 19.518
  02:01:28 STEP 1828 H= 457:00 +CPU= 19.953
  02:01:48 STEP 1829 H= 457:15 +CPU= 19.562
  02:02:08 STEP 1830 H= 457:30 +CPU= 19.714
  02:02:27 STEP 1831 H= 457:45 +CPU= 19.451
  02:02:47 STEP 1832 H= 458:00 +CPU= 19.920
  02:03:06 STEP 1833 H= 458:15 +CPU= 19.274
  02:03:26 STEP 1834 H= 458:30 +CPU= 19.208
  02:03:45 STEP 1835 H= 458:45 +CPU= 19.571
  02:04:11 STEP 1836 H= 459:00 +CPU= 25.707
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001836
  02:04:30 STEP 1837 H= 459:15 +CPU= 19.283
  02:04:50 STEP 1838 H= 459:30 +CPU= 19.346
  02:05:09 STEP 1839 H= 459:45 +CPU= 19.545
  02:05:30 STEP 1840 H= 460:00 +CPU= 20.755
  02:05:50 STEP 1841 H= 460:15 +CPU= 19.574
  02:06:09 STEP 1842 H= 460:30 +CPU= 19.611
  02:06:29 STEP 1843 H= 460:45 +CPU= 19.675
  02:06:49 STEP 1844 H= 461:00 +CPU= 20.357
  02:07:09 STEP 1845 H= 461:15 +CPU= 19.496
  02:07:28 STEP 1846 H= 461:30 +CPU= 19.377
  02:07:48 STEP 1847 H= 461:45 +CPU= 19.346
  02:08:13 STEP 1848 H= 462:00 +CPU= 25.576
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001848
  02:08:33 STEP 1849 H= 462:15 +CPU= 19.648
  02:08:53 STEP 1850 H= 462:30 +CPU= 19.461
  02:09:12 STEP 1851 H= 462:45 +CPU= 19.598
  02:09:33 STEP 1852 H= 463:00 +CPU= 20.241
  02:09:52 STEP 1853 H= 463:15 +CPU= 19.684
  02:10:12 STEP 1854 H= 463:30 +CPU= 19.283
  02:10:32 STEP 1855 H= 463:45 +CPU= 20.246
  02:10:52 STEP 1856 H= 464:00 +CPU= 20.380
  02:11:11 STEP 1857 H= 464:15 +CPU= 19.170
  02:11:31 STEP 1858 H= 464:30 +CPU= 19.370
  02:11:51 STEP 1859 H= 464:45 +CPU= 19.906
  02:12:17 STEP 1860 H= 465:00 +CPU= 25.638
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001860
  02:12:36 STEP 1861 H= 465:15 +CPU= 19.516
  02:12:56 STEP 1862 H= 465:30 +CPU= 19.473
  02:13:15 STEP 1863 H= 465:45 +CPU= 19.632
  02:13:35 STEP 1864 H= 466:00 +CPU= 19.879
  02:13:54 STEP 1865 H= 466:15 +CPU= 19.185
  02:14:14 STEP 1866 H= 466:30 +CPU= 19.819
  02:14:35 STEP 1867 H= 466:45 +CPU= 20.197
  02:14:55 STEP 1868 H= 467:00 +CPU= 20.421
  02:15:14 STEP 1869 H= 467:15 +CPU= 19.473
  02:15:34 STEP 1870 H= 467:30 +CPU= 19.908
  02:15:54 STEP 1871 H= 467:45 +CPU= 19.965
  02:16:22 STEP 1872 H= 468:00 +CPU= 27.217
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001872
  02:16:41 STEP 1873 H= 468:15 +CPU= 19.294
  02:17:01 STEP 1874 H= 468:30 +CPU= 20.062
  02:17:21 STEP 1875 H= 468:45 +CPU= 20.121
  02:17:42 STEP 1876 H= 469:00 +CPU= 20.227
  02:18:01 STEP 1877 H= 469:15 +CPU= 19.562
  02:18:21 STEP 1878 H= 469:30 +CPU= 19.500
  02:18:40 STEP 1879 H= 469:45 +CPU= 19.731
  02:19:01 STEP 1880 H= 470:00 +CPU= 20.807
  02:19:21 STEP 1881 H= 470:15 +CPU= 19.274
  02:19:40 STEP 1882 H= 470:30 +CPU= 19.712
  02:20:00 STEP 1883 H= 470:45 +CPU= 19.523
  02:20:26 STEP 1884 H= 471:00 +CPU= 26.124
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001884
  02:20:46 STEP 1885 H= 471:15 +CPU= 19.622
  02:21:05 STEP 1886 H= 471:30 +CPU= 19.326
  02:21:24 STEP 1887 H= 471:45 +CPU= 19.363
  02:21:46 STEP 1888 H= 472:00 +CPU= 20.999
  02:22:06 STEP 1889 H= 472:15 +CPU= 20.034
  02:22:25 STEP 1890 H= 472:30 +CPU= 19.363
  02:22:44 STEP 1891 H= 472:45 +CPU= 19.416
  02:23:05 STEP 1892 H= 473:00 +CPU= 20.698
  02:23:25 STEP 1893 H= 473:15 +CPU= 19.528
  02:23:44 STEP 1894 H= 473:30 +CPU= 19.313
  02:24:03 STEP 1895 H= 473:45 +CPU= 19.120
  02:24:29 STEP 1896 H= 474:00 +CPU= 25.657
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001896
  02:24:49 STEP 1897 H= 474:15 +CPU= 20.026
  02:25:09 STEP 1898 H= 474:30 +CPU= 19.738
  02:25:29 STEP 1899 H= 474:45 +CPU= 19.891
  02:25:49 STEP 1900 H= 475:00 +CPU= 20.346
  02:26:09 STEP 1901 H= 475:15 +CPU= 19.545
  02:26:28 STEP 1902 H= 475:30 +CPU= 19.419
  02:26:48 STEP 1903 H= 475:45 +CPU= 20.148
  02:27:09 STEP 1904 H= 476:00 +CPU= 20.486
  02:27:28 STEP 1905 H= 476:15 +CPU= 19.418
  02:27:48 STEP 1906 H= 476:30 +CPU= 19.454
  02:28:07 STEP 1907 H= 476:45 +CPU= 19.464
  02:28:33 STEP 1908 H= 477:00 +CPU= 25.644
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001908
  02:28:53 STEP 1909 H= 477:15 +CPU= 19.957
  02:29:13 STEP 1910 H= 477:30 +CPU= 20.112
  02:29:33 STEP 1911 H= 477:45 +CPU= 20.335
  02:29:54 STEP 1912 H= 478:00 +CPU= 20.685
  02:30:14 STEP 1913 H= 478:15 +CPU= 19.874
  02:30:34 STEP 1914 H= 478:30 +CPU= 19.764
  02:30:54 STEP 1915 H= 478:45 +CPU= 19.982
  02:31:15 STEP 1916 H= 479:00 +CPU= 20.690
  02:31:35 STEP 1917 H= 479:15 +CPU= 19.824
  02:31:54 STEP 1918 H= 479:30 +CPU= 19.585
  02:32:14 STEP 1919 H= 479:45 +CPU= 19.492
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a3cr
(argv4) batchid: 1018
(argv5) wuid: 12292674
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_042aa3a05d002afae763a06085c2491d to: /var/lib/boinc/slots/1/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674.zip
ic_ancil_file: ic_ancil_12292674
ifsdata_file: ifsdata_12292674
climate_data_file: clim_data_12292674
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/1/ic_ancil_12292674.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12292674.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_088e67a42cd2dfccbf299ef0e824d68c to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12292674.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12292674.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12292674.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12292674.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12292674.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12292674.xml
-- Model is restarting --
Adjusting last_iter, 1920, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12292674.xml
last_cpu_time: 38215
upload_file_number: 19
last_iter: 1921
last_upload: 1642500
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0_r1419436944
The child process has been launched with process id: 568595
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:Happywetter11:1:1:568595:568595] [20240614:023328:1718325208:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  02:33:49 STEP 1920 H= 480:00 +CPU= 19.007
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

I      I Hor. Flux of PEII      0.00001 I              I              I              I               I
I      I Vert.Flux of PEII      0.00000 I              I              I              I               I
I      I Work Pres.G.F  II              I      0.44899 I              I              I               I
I      I Conv. Ad.-Int. II     -0.44915 I              I              I              I               I
DDH--------------------------------------------------------------------------------------------------I
I DYN- Horizont. Flux   II      0.12209 I      0.01501 I      1.97095 I      0.01186 I     -0.03169  I
I    I Vertical Flux    II      0.00000 I      0.00000 I      0.00000 I      0.00000 I      0.00000  I
DDH--------------------------------------------------------------------------------------------------I
  
DDH----------------------------------------------------------------------------------------------------------------------------------
  I   - Mass balance    (10**(-6)  KG/M**2) 
  
  I INITIAL      I HORIZ. FLUX  I DYN. FLUX    I DYN. FLUX    I PHYS. FLUX   I PHYS. FLUX   I CURRENT      I I I RESIDUAL     I
  I MASS         I              I TOA          I SURFACE      I TOA          I SURFACE      I MASS         I I I              I
  ---------------------------------------------------------------------------------------------------------- I ----------------
  -              -              -              -              -              -              -              - I -              -
  - 10341776742. I          21. .           0. .           0. .           0. .           0. I 10342469172. - I -      692408. -
  -              -              -              -              -              -              -              - I -              -
  ---------------------------------------------------------------------------------------------------------- I ----------------
 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.7785371161E-21      I       0.2497205796E-05      I
ICloud fractionI      0.0000000000E+00      I       0.6906884468E-01      I
I  Relat. Hum. I      0.1386309797E+00      I       0.3785901261E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1067950656E+04      I
DDH------------------------------------------------------------------------
  
 02:33:49 STEP 1920 H= 480:00 +CPU= 19.007
 END CNT3
 NSTEP =  1920 CNT0   000000000  18.864  18.864  18.897   0.998               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN            0/           0
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1      3    0       46      1044068        0            0     1.2     1.2     0.0     1.5     0.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1   21486.5   21486.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1   19039.9   19039.9     88.61      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                    1    2503.5    2503.5     11.65      0.00
  10 SCAN2M - PHYSICS                                1   11969.1   11969.1     55.71      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  1      28.5      28.5      0.13      0.00
  28 RESTART FILES - READING                         1    2046.2    2046.2      9.52      0.00
  31 POSDDH IN STEPO                                 1       0.8       0.8      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                      1    3603.7    3603.7     16.77      0.00
  39 SU0YOMB                                         1     280.7     280.7      1.31      0.00
  51 SCAN2M   - SL COMM. PART 1                      1       7.2       7.2      0.03      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  1       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM        4     119.7     119.7      2.23      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM         4      50.4      50.4      0.94      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      32.9      32.9      0.15      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION              4       7.9       7.9      0.15      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION              4      20.4      20.4      0.38      0.00
 400 GSTATS                                       1119       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                   987       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME       21.5 CPU TIME      21.4 VECTOR TIME       21.4


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY    90756.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc/slots/1/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/1/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+000384
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292674/ICMUAh7zg+001896
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0_r1419436944_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1728900
02:37:31 (568592): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Jun 2024 00:36:20 1504810 22446283 oifs_43r3_bl_a3cr_2016092300_20_1018_12292674_0 1,728,900 38,236 0.0221


©2024 climateprediction.net