climateprediction.net home page
Task 22302868

Task 22302868

Name oifs_43r3_ps_0104_2012050100_123_981_12197748_2
Workunit 12197748
Created 24 Jan 2023, 10:28:27 UTC
Sent 24 Jan 2023, 10:36:40 UTC
Report deadline 25 Mar 2023, 10:36:40 UTC
Received 25 Jan 2023, 15:19:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1517479
Run time 16 hours 18 min 24 sec
CPU time 16 hours 14 min 26 sec
Validate state Invalid
Credit 9,313.92
Device peak FLOPS 4.36 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.68 MB
Peak swap size 4,978.14 MB
Peak disk usage 7,976.89 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
87 H=1487:00 +CPU= 21.758
  02:53:09 STEP 1488 H=1488:00 +CPU= 41.874
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001488
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001488
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001464
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001464
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001464
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001476
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001476
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001476
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_61.zip
Uploading the intermediate file: upload_file_61.zip
  02:53:32 STEP 1489 H=1489:00 +CPU= 23.601
  02:53:54 STEP 1490 H=1490:00 +CPU= 22.196
  02:54:29 STEP 1491 H=1491:00 +CPU= 35.011
  02:54:51 STEP 1492 H=1492:00 +CPU= 21.749
  02:55:12 STEP 1493 H=1493:00 +CPU= 20.757
  02:55:46 STEP 1494 H=1494:00 +CPU= 33.550
  02:56:06 STEP 1495 H=1495:00 +CPU= 20.559
  02:56:31 STEP 1496 H=1496:00 +CPU= 24.428
  02:57:04 STEP 1497 H=1497:00 +CPU= 33.020
  02:57:25 STEP 1498 H=1498:00 +CPU= 20.836
  02:57:45 STEP 1499 H=1499:00 +CPU= 20.758
  02:58:22 STEP 1500 H=1500:00 +CPU= 36.797
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001500
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001500
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001500
  02:58:44 STEP 1501 H=1501:00 +CPU= 21.680
  02:59:06 STEP 1502 H=1502:00 +CPU= 21.832
  02:59:39 STEP 1503 H=1503:00 +CPU= 33.298
  03:00:01 STEP 1504 H=1504:00 +CPU= 21.877
  03:00:22 STEP 1505 H=1505:00 +CPU= 20.978
  03:00:56 STEP 1506 H=1506:00 +CPU= 33.890
  03:01:16 STEP 1507 H=1507:00 +CPU= 20.374
  03:01:38 STEP 1508 H=1508:00 +CPU= 21.293
  03:02:12 STEP 1509 H=1509:00 +CPU= 34.187
  03:02:37 STEP 1510 H=1510:00 +CPU= 24.850
  03:02:59 STEP 1511 H=1511:00 +CPU= 22.069
  03:03:38 STEP 1512 H=1512:00 +CPU= 39.123
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001512
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001512
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001500
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001500
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001500
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_62.zip
Uploading the intermediate file: upload_file_62.zip
  03:03:59 STEP 1513 H=1513:00 +CPU= 21.269
  03:04:21 STEP 1514 H=1514:00 +CPU= 21.885
  03:04:57 STEP 1515 H=1515:00 +CPU= 35.886
  03:05:19 STEP 1516 H=1516:00 +CPU= 21.734
  03:05:40 STEP 1517 H=1517:00 +CPU= 21.200
  03:06:15 STEP 1518 H=1518:00 +CPU= 34.671
  03:06:36 STEP 1519 H=1519:00 +CPU= 21.079
  03:06:58 STEP 1520 H=1520:00 +CPU= 21.875
  03:07:33 STEP 1521 H=1521:00 +CPU= 34.970
  03:07:56 STEP 1522 H=1522:00 +CPU= 23.081
  03:08:16 STEP 1523 H=1523:00 +CPU= 20.438
  03:08:57 STEP 1524 H=1524:00 +CPU= 40.894
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001524
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001524
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001524
  03:09:20 STEP 1525 H=1525:00 +CPU= 22.567
  03:09:42 STEP 1526 H=1526:00 +CPU= 21.775
  03:10:15 STEP 1527 H=1527:00 +CPU= 33.038
  03:10:36 STEP 1528 H=1528:00 +CPU= 21.458
  03:10:57 STEP 1529 H=1529:00 +CPU= 20.757
  03:11:32 STEP 1530 H=1530:00 +CPU= 34.655
  03:11:53 STEP 1531 H=1531:00 +CPU= 21.265
  03:12:15 STEP 1532 H=1532:00 +CPU= 21.982
  03:12:49 STEP 1533 H=1533:00 +CPU= 33.426
  03:13:11 STEP 1534 H=1534:00 +CPU= 22.360
  03:13:34 STEP 1535 H=1535:00 +CPU= 22.835
  03:14:13 STEP 1536 H=1536:00 +CPU= 39.330
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001536
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001536
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001524
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001524
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001524
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_63.zip
Uploading the intermediate file: upload_file_63.zip
  03:14:35 STEP 1537 H=1537:00 +CPU= 21.243
  03:14:56 STEP 1538 H=1538:00 +CPU= 21.295
  03:15:29 STEP 1539 H=1539:00 +CPU= 33.328
  03:15:50 STEP 1540 H=1540:00 +CPU= 21.044
  03:16:11 STEP 1541 H=1541:00 +CPU= 20.173
  03:16:45 STEP 1542 H=1542:00 +CPU= 34.861
  03:17:06 STEP 1543 H=1543:00 +CPU= 21.051
  03:17:28 STEP 1544 H=1544:00 +CPU= 21.739
  03:18:02 STEP 1545 H=1545:00 +CPU= 33.384
  03:18:23 STEP 1546 H=1546:00 +CPU= 21.487
  03:18:44 STEP 1547 H=1547:00 +CPU= 20.681
  03:19:20 STEP 1548 H=1548:00 +CPU= 36.247
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001548
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001548
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001548
  03:19:41 STEP 1549 H=1549:00 +CPU= 20.694
  03:20:02 STEP 1550 H=1550:00 +CPU= 20.827
  03:20:35 STEP 1551 H=1551:00 +CPU= 33.228
  03:20:56 STEP 1552 H=1552:00 +CPU= 21.205
  03:21:16 STEP 1553 H=1553:00 +CPU= 19.529
  03:21:53 STEP 1554 H=1554:00 +CPU= 37.241
  03:22:17 STEP 1555 H=1555:00 +CPU= 23.235
  03:22:41 STEP 1556 H=1556:00 +CPU= 23.907
  03:23:18 STEP 1557 H=1557:00 +CPU= 36.266
  03:23:43 STEP 1558 H=1558:00 +CPU= 24.363
  03:24:07 STEP 1559 H=1559:00 +CPU= 24.509
  03:24:51 STEP 1560 H=1560:00 +CPU= 44.024
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001560
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001560
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001548
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001548
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001548
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_64.zip
Uploading the intermediate file: upload_file_64.zip
  03:25:15 STEP 1561 H=1561:00 +CPU= 23.745
  03:25:39 STEP 1562 H=1562:00 +CPU= 24.137
  03:26:17 STEP 1563 H=1563:00 +CPU= 37.268
  03:26:41 STEP 1564 H=1564:00 +CPU= 24.390
  03:27:04 STEP 1565 H=1565:00 +CPU= 23.380
  03:27:43 STEP 1566 H=1566:00 +CPU= 38.083
  03:28:06 STEP 1567 H=1567:00 +CPU= 22.956
  03:28:29 STEP 1568 H=1568:00 +CPU= 23.809
  03:29:06 STEP 1569 H=1569:00 +CPU= 36.139
  03:29:29 STEP 1570 H=1570:00 +CPU= 23.735
  03:29:52 STEP 1571 H=1571:00 +CPU= 23.058
  03:30:33 STEP 1572 H=1572:00 +CPU= 40.306
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001572
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001572
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001572
  03:30:56 STEP 1573 H=1573:00 +CPU= 23.044
  03:31:20 STEP 1574 H=1574:00 +CPU= 23.814
  03:31:56 STEP 1575 H=1575:00 +CPU= 36.201
  03:32:20 STEP 1576 H=1576:00 +CPU= 23.888
  03:32:43 STEP 1577 H=1577:00 +CPU= 23.034
  03:33:23 STEP 1578 H=1578:00 +CPU= 40.107
  03:33:48 STEP 1579 H=1579:00 +CPU= 25.277
  03:34:14 STEP 1580 H=1580:00 +CPU= 25.565
  03:34:54 STEP 1581 H=1581:00 +CPU= 39.636
  03:35:19 STEP 1582 H=1582:00 +CPU= 24.816
  03:35:42 STEP 1583 H=1583:00 +CPU= 23.742
  03:36:27 STEP 1584 H=1584:00 +CPU= 44.153
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001584
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001584
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001572
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001572
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001572
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_65.zip
Uploading the intermediate file: upload_file_65.zip
  03:36:50 STEP 1585 H=1585:00 +CPU= 23.206
  03:37:14 STEP 1586 H=1586:00 +CPU= 23.907
  03:37:51 STEP 1587 H=1587:00 +CPU= 36.263
  03:38:15 STEP 1588 H=1588:00 +CPU= 24.359
  03:38:41 STEP 1589 H=1589:00 +CPU= 25.553
  03:39:21 STEP 1590 H=1590:00 +CPU= 40.496
  03:39:45 STEP 1591 H=1591:00 +CPU= 23.980
  03:40:10 STEP 1592 H=1592:00 +CPU= 24.983
  03:40:48 STEP 1593 H=1593:00 +CPU= 38.077
  03:41:13 STEP 1594 H=1594:00 +CPU= 24.958
  03:41:38 STEP 1595 H=1595:00 +CPU= 24.081
  03:42:20 STEP 1596 H=1596:00 +CPU= 42.809
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001596
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001596
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001596
  03:42:45 STEP 1597 H=1597:00 +CPU= 24.190
  03:43:10 STEP 1598 H=1598:00 +CPU= 24.867
  03:43:48 STEP 1599 H=1599:00 +CPU= 37.895
  03:44:12 STEP 1600 H=1600:00 +CPU= 23.949
  03:44:35 STEP 1601 H=1601:00 +CPU= 22.992
  03:45:14 STEP 1602 H=1602:00 +CPU= 39.006
  03:45:40 STEP 1603 H=1603:00 +CPU= 24.976
  03:46:04 STEP 1604 H=1604:00 +CPU= 23.840
  03:46:40 STEP 1605 H=1605:00 +CPU= 36.318
  03:47:04 STEP 1606 H=1606:00 +CPU= 23.793
  03:47:27 STEP 1607 H=1607:00 +CPU= 23.662
  03:48:12 STEP 1608 H=1608:00 +CPU= 44.438
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001608
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001608
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001596
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001596
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001596
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_66.zip
Uploading the intermediate file: upload_file_66.zip
  03:48:36 STEP 1609 H=1609:00 +CPU= 23.639
  03:49:01 STEP 1610 H=1610:00 +CPU= 24.589
  03:49:38 STEP 1611 H=1611:00 +CPU= 37.490
  03:50:03 STEP 1612 H=1612:00 +CPU= 24.466
  03:50:29 STEP 1613 H=1613:00 +CPU= 25.380
  03:51:09 STEP 1614 H=1614:00 +CPU= 40.600
  03:51:35 STEP 1615 H=1615:00 +CPU= 25.195
  03:52:01 STEP 1616 H=1616:00 +CPU= 26.303
  03:52:41 STEP 1617 H=1617:00 +CPU= 39.512
  03:53:07 STEP 1618 H=1618:00 +CPU= 26.064
  03:53:32 STEP 1619 H=1619:00 +CPU= 25.491
  03:54:16 STEP 1620 H=1620:00 +CPU= 43.807
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001620
  03:54:41 STEP 1621 H=1621:00 +CPU= 25.172
  03:55:07 STEP 1622 H=1622:00 +CPU= 25.708
  03:55:44 STEP 1623 H=1623:00 +CPU= 36.141
  03:56:08 STEP 1624 H=1624:00 +CPU= 24.300
  03:56:32 STEP 1625 H=1625:00 +CPU= 23.724
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0104
(argv4) batchid: 981
(argv5) wuid: 12197748
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mydisks/a/boinc_lib/slots/14
Project directory is: /mydisks/a/boinc_lib/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748
..mkdir for temp folder for results failed
Copying: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c31547488b85abd7dee5be9fa77b1afd to: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_0104_2012050100_123_981_12197748.zip
Unzipping the namelist zip file: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_0104_2012050100_123_981_12197748.zip
ic_ancil_file: ic_ancil_12197748
ifsdata_file: ifsdata_12197748
climate_data_file: clim_data_12197748
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f761aeb0b9f4f6620269cb483a6118dc to: /mydisks/a/boinc_lib/slots/14/ic_ancil_12197748.zip
Unzipping the IC ancils zip file: /mydisks/a/boinc_lib/slots/14/ic_ancil_12197748.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec0332c27bf4e9a6bb5decbc16e70db6 to: /mydisks/a/boinc_lib/slots/14/ifsdata/ifsdata_12197748.zip
Unzipping the ifsdata_zip file: /mydisks/a/boinc_lib/slots/14/ifsdata/ifsdata_12197748.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/a/boinc_lib/slots/14/159l_2/clim_data_12197748.zip
Unzipping the climate data zip file: /mydisks/a/boinc_lib/slots/14/159l_2/clim_data_12197748.zip
Checking for progress XML file: /mydisks/a/boinc_lib/slots/14/progress_file_12197748.xml
Opened progress file ok : /mydisks/a/boinc_lib/slots/14/progress_file_12197748.xml
-- Model is restarting --
Adjusting last_iter, 1626, to previous model restart step.
Creating progress file: /mydisks/a/boinc_lib/slots/14/progress_file_12197748.xml
last_cpu_time: 49801
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431
The child process has been launched with process id: 8109
Executing the command: /mydisks/a/boinc_lib/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8109:8109] [20230125:040439:1674648279:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:05:20 STEP 1608 H=1608:00 +CPU= 38.420
  04:05:42 STEP 1609 H=1609:00 +CPU= 21.332
  04:06:03 STEP 1610 H=1610:00 +CPU= 21.541
  04:06:40 STEP 1611 H=1611:00 +CPU= 36.783
  04:07:03 STEP 1612 H=1612:00 +CPU= 23.054
  04:07:24 STEP 1613 H=1613:00 +CPU= 20.750
  04:07:59 STEP 1614 H=1614:00 +CPU= 34.810
  04:08:23 STEP 1615 H=1615:00 +CPU= 24.080
  04:08:46 STEP 1616 H=1616:00 +CPU= 23.218
  04:09:20 STEP 1617 H=1617:00 +CPU= 33.560
  04:09:42 STEP 1618 H=1618:00 +CPU= 22.003
  04:10:03 STEP 1619 H=1619:00 +CPU= 21.026
  04:10:42 STEP 1620 H=1620:00 +CPU= 38.833
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001620
  04:11:02 STEP 1621 H=1621:00 +CPU= 20.863
  04:11:24 STEP 1622 H=1622:00 +CPU= 21.843
  04:12:01 STEP 1623 H=1623:00 +CPU= 36.537
  04:12:23 STEP 1624 H=1624:00 +CPU= 21.596
  04:12:43 STEP 1625 H=1625:00 +CPU= 20.783
  04:13:17 STEP 1626 H=1626:00 +CPU= 33.919
  04:13:39 STEP 1627 H=1627:00 +CPU= 21.308
  04:14:00 STEP 1628 H=1628:00 +CPU= 21.232
  04:14:33 STEP 1629 H=1629:00 +CPU= 33.146
  04:14:54 STEP 1630 H=1630:00 +CPU= 21.315
  04:15:18 STEP 1631 H=1631:00 +CPU= 23.074
  04:15:56 STEP 1632 H=1632:00 +CPU= 38.666
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001632
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001632
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001620
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001620
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001620
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_67.zip
Uploading the intermediate file: upload_file_67.zip
  04:16:17 STEP 1633 H=1633:00 +CPU= 20.902
  04:16:39 STEP 1634 H=1634:00 +CPU= 21.361
  04:17:12 STEP 1635 H=1635:00 +CPU= 33.034
  04:17:34 STEP 1636 H=1636:00 +CPU= 21.987
  04:17:54 STEP 1637 H=1637:00 +CPU= 20.422
  04:18:28 STEP 1638 H=1638:00 +CPU= 33.869
  04:18:50 STEP 1639 H=1639:00 +CPU= 21.597
  04:19:12 STEP 1640 H=1640:00 +CPU= 21.803
  04:19:46 STEP 1641 H=1641:00 +CPU= 33.764
  04:20:09 STEP 1642 H=1642:00 +CPU= 23.159
  04:20:30 STEP 1643 H=1643:00 +CPU= 21.176
  04:21:08 STEP 1644 H=1644:00 +CPU= 37.772
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001644
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001644
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001644
  04:21:29 STEP 1645 H=1645:00 +CPU= 20.635
  04:21:51 STEP 1646 H=1646:00 +CPU= 22.280
  04:22:24 STEP 1647 H=1647:00 +CPU= 33.234
  04:22:45 STEP 1648 H=1648:00 +CPU= 20.972
  04:23:05 STEP 1649 H=1649:00 +CPU= 20.131
  04:23:39 STEP 1650 H=1650:00 +CPU= 33.628
  04:24:00 STEP 1651 H=1651:00 +CPU= 20.678
  04:24:18 STEP 1652 H=1652:00 +CPU= 18.507
  04:24:48 STEP 1653 H=1653:00 +CPU= 29.284
  04:25:09 STEP 1654 H=1654:00 +CPU= 21.756
  04:25:31 STEP 1655 H=1655:00 +CPU= 21.619
  04:26:15 STEP 1656 H=1656:00 +CPU= 43.552
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001656
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001656
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001644
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001644
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001644
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_68.zip
Uploading the intermediate file: upload_file_68.zip
  04:26:36 STEP 1657 H=1657:00 +CPU= 20.924
  04:26:58 STEP 1658 H=1658:00 +CPU= 21.973
  04:27:31 STEP 1659 H=1659:00 +CPU= 33.369
  04:27:52 STEP 1660 H=1660:00 +CPU= 21.013
  04:28:12 STEP 1661 H=1661:00 +CPU= 20.198
  04:28:46 STEP 1662 H=1662:00 +CPU= 33.679
  04:29:09 STEP 1663 H=1663:00 +CPU= 23.407
  04:29:32 STEP 1664 H=1664:00 +CPU= 22.808
  04:30:06 STEP 1665 H=1665:00 +CPU= 33.245
  04:30:27 STEP 1666 H=1666:00 +CPU= 21.075
  04:30:47 STEP 1667 H=1667:00 +CPU= 20.180
  04:31:28 STEP 1668 H=1668:00 +CPU= 41.102
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001668
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001668
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001668
  04:31:52 STEP 1669 H=1669:00 +CPU= 24.132
  04:32:14 STEP 1670 H=1670:00 +CPU= 21.502
  04:32:47 STEP 1671 H=1671:00 +CPU= 33.138
  04:33:08 STEP 1672 H=1672:00 +CPU= 20.695
  04:33:27 STEP 1673 H=1673:00 +CPU= 19.370
  04:33:53 STEP 1674 H=1674:00 +CPU= 26.203
  04:34:09 STEP 1675 H=1675:00 +CPU= 15.452
  04:34:26 STEP 1676 H=1676:00 +CPU= 16.956
  04:35:00 STEP 1677 H=1677:00 +CPU= 33.914
  04:35:22 STEP 1678 H=1678:00 +CPU= 22.036
  04:35:43 STEP 1679 H=1679:00 +CPU= 21.568
  04:36:25 STEP 1680 H=1680:00 +CPU= 41.669
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001680
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001680
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001668
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001668
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001668
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_69.zip
Uploading the intermediate file: upload_file_69.zip
  04:36:46 STEP 1681 H=1681:00 +CPU= 21.270
  04:37:08 STEP 1682 H=1682:00 +CPU= 21.499
  04:37:41 STEP 1683 H=1683:00 +CPU= 33.425
  04:38:05 STEP 1684 H=1684:00 +CPU= 23.266
  04:38:27 STEP 1685 H=1685:00 +CPU= 22.177
  04:39:03 STEP 1686 H=1686:00 +CPU= 35.549
  04:39:24 STEP 1687 H=1687:00 +CPU= 21.509
  04:39:46 STEP 1688 H=1688:00 +CPU= 21.753
  04:40:21 STEP 1689 H=1689:00 +CPU= 34.785
  04:40:42 STEP 1690 H=1690:00 +CPU= 21.636
  04:41:04 STEP 1691 H=1691:00 +CPU= 21.133
  04:41:41 STEP 1692 H=1692:00 +CPU= 37.500
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001692
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001692
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001692
  04:42:03 STEP 1693 H=1693:00 +CPU= 21.520
  04:42:24 STEP 1694 H=1694:00 +CPU= 21.440
  04:42:58 STEP 1695 H=1695:00 +CPU= 33.557
  04:43:20 STEP 1696 H=1696:00 +CPU= 21.885
  04:43:41 STEP 1697 H=1697:00 +CPU= 21.077
  04:44:16 STEP 1698 H=1698:00 +CPU= 35.021
  04:44:37 STEP 1699 H=1699:00 +CPU= 21.341
  04:44:59 STEP 1700 H=1700:00 +CPU= 22.058
  04:45:39 STEP 1701 H=1701:00 +CPU= 39.282
  04:46:02 STEP 1702 H=1702:00 +CPU= 22.738
  04:46:24 STEP 1703 H=1703:00 +CPU= 22.365
  04:47:04 STEP 1704 H=1704:00 +CPU= 40.105
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001704
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001704
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001692
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001692
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001692
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_70.zip
Uploading the intermediate file: upload_file_70.zip
  04:47:26 STEP 1705 H=1705:00 +CPU= 21.342
  04:47:47 STEP 1706 H=1706:00 +CPU= 21.501
  04:48:21 STEP 1707 H=1707:00 +CPU= 33.876
  04:48:43 STEP 1708 H=1708:00 +CPU= 22.336
  04:49:05 STEP 1709 H=1709:00 +CPU= 21.414
  04:49:39 STEP 1710 H=1710:00 +CPU= 34.647
  04:50:01 STEP 1711 H=1711:00 +CPU= 21.308
  04:50:23 STEP 1712 H=1712:00 +CPU= 21.800
  04:50:57 STEP 1713 H=1713:00 +CPU= 34.030
  04:51:20 STEP 1714 H=1714:00 +CPU= 23.062
  04:51:44 STEP 1715 H=1715:00 +CPU= 23.935
  04:52:25 STEP 1716 H=1716:00 +CPU= 41.330
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001716
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001716
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001716
  04:52:47 STEP 1717 H=1717:00 +CPU= 22.162
  04:53:10 STEP 1718 H=1718:00 +CPU= 22.811
  04:53:45 STEP 1719 H=1719:00 +CPU= 34.992
  04:54:07 STEP 1720 H=1720:00 +CPU= 21.541
  04:54:28 STEP 1721 H=1721:00 +CPU= 20.830
  04:55:02 STEP 1722 H=1722:00 +CPU= 34.479
  04:55:24 STEP 1723 H=1723:00 +CPU= 21.308
  04:55:46 STEP 1724 H=1724:00 +CPU= 21.992
  04:56:20 STEP 1725 H=1725:00 +CPU= 34.248
  04:56:42 STEP 1726 H=1726:00 +CPU= 22.231
  04:57:04 STEP 1727 H=1727:00 +CPU= 21.727
  04:57:44 STEP 1728 H=1728:00 +CPU= 40.169
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001728
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001728
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001716
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001716
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001716
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_71.zip
Uploading the intermediate file: upload_file_71.zip
  04:58:05 STEP 1729 H=1729:00 +CPU= 21.307
  04:58:28 STEP 1730 H=1730:00 +CPU= 22.034
  04:59:01 STEP 1731 H=1731:00 +CPU= 33.870
  04:59:23 STEP 1732 H=1732:00 +CPU= 21.584
  04:59:47 STEP 1733 H=1733:00 +CPU= 23.847
  05:00:23 STEP 1734 H=1734:00 +CPU= 36.277
  05:00:46 STEP 1735 H=1735:00 +CPU= 22.685
  05:01:10 STEP 1736 H=1736:00 +CPU= 24.018
  05:01:44 STEP 1737 H=1737:00 +CPU= 34.344
  05:02:09 STEP 1738 H=1738:00 +CPU= 24.525
  05:02:32 STEP 1739 H=1739:00 +CPU= 23.354
  05:03:11 STEP 1740 H=1740:00 +CPU= 38.239
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001740
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001740
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001740
  05:03:33 STEP 1741 H=1741:00 +CPU= 22.320
  05:03:55 STEP 1742 H=1742:00 +CPU= 22.185
  05:04:29 STEP 1743 H=1743:00 +CPU= 33.772
  05:04:51 STEP 1744 H=1744:00 +CPU= 22.374
  05:05:14 STEP 1745 H=1745:00 +CPU= 22.084
  05:05:49 STEP 1746 H=1746:00 +CPU= 35.233
  05:06:13 STEP 1747 H=1747:00 +CPU= 24.390
  05:06:38 STEP 1748 H=1748:00 +CPU= 24.945
  05:07:13 STEP 1749 H=1749:00 +CPU= 34.937
  05:07:38 STEP 1750 H=1750:00 +CPU= 24.419
  05:08:01 STEP 1751 H=1751:00 +CPU= 23.313
  05:08:42 STEP 1752 H=1752:00 +CPU= 40.517
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001752
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001752
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001740
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001740
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001740
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_72.zip
Uploading the intermediate file: upload_file_72.zip
  05:09:05 STEP 1753 H=1753:00 +CPU= 22.900
  05:09:26 STEP 1754 H=1754:00 +CPU= 21.825
  05:10:01 STEP 1755 H=1755:00 +CPU= 34.198
  05:10:24 STEP 1756 H=1756:00 +CPU= 23.457
  05:10:46 STEP 1757 H=1757:00 +CPU= 21.468
  05:11:21 STEP 1758 H=1758:00 +CPU= 35.351
  05:11:43 STEP 1759 H=1759:00 +CPU= 22.301
  05:12:08 STEP 1760 H=1760:00 +CPU= 24.210
  05:12:42 STEP 1761 H=1761:00 +CPU= 34.320
  05:13:04 STEP 1762 H=1762:00 +CPU= 21.757
  05:13:25 STEP 1763 H=1763:00 +CPU= 21.313
  05:14:04 STEP 1764 H=1764:00 +CPU= 39.169
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001764
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001764
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001764
  05:14:27 STEP 1765 H=1765:00 +CPU= 22.710
  05:14:51 STEP 1766 H=1766:00 +CPU= 23.972
  05:15:25 STEP 1767 H=1767:00 +CPU= 33.582
  05:15:47 STEP 1768 H=1768:00 +CPU= 21.781
  05:16:08 STEP 1769 H=1769:00 +CPU= 21.447
  05:16:43 STEP 1770 H=1770:00 +CPU= 35.297
  05:17:07 STEP 1771 H=1771:00 +CPU= 23.662
  05:17:29 STEP 1772 H=1772:00 +CPU= 22.184
  05:18:03 STEP 1773 H=1773:00 +CPU= 33.941
  05:18:25 STEP 1774 H=1774:00 +CPU= 21.995
  05:18:48 STEP 1775 H=1775:00 +CPU= 22.576
  05:19:30 STEP 1776 H=1776:00 +CPU= 41.570
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001776
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001776
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001764
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001764
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001764
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_73.zip
Uploading the intermediate file: upload_file_73.zip
  05:19:51 STEP 1777 H=1777:00 +CPU= 21.286
  05:20:13 STEP 1778 H=1778:00 +CPU= 21.804
  05:20:48 STEP 1779 H=1779:00 +CPU= 34.797
  05:21:10 STEP 1780 H=1780:00 +CPU= 22.604
  05:21:33 STEP 1781 H=1781:00 +CPU= 22.230
  05:22:09 STEP 1782 H=1782:00 +CPU= 36.902
  05:22:33 STEP 1783 H=1783:00 +CPU= 23.182
  05:22:56 STEP 1784 H=1784:00 +CPU= 23.248
  05:23:30 STEP 1785 H=1785:00 +CPU= 34.236
  05:23:52 STEP 1786 H=1786:00 +CPU= 22.152
  05:24:15 STEP 1787 H=1787:00 +CPU= 22.538
  05:24:53 STEP 1788 H=1788:00 +CPU= 38.171
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001788
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001788
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001788
  05:25:14 STEP 1789 H=1789:00 +CPU= 20.980
  05:25:36 STEP 1790 H=1790:00 +CPU= 22.140
  05:26:11 STEP 1791 H=1791:00 +CPU= 34.909
  05:26:36 STEP 1792 H=1792:00 +CPU= 24.933
  05:26:58 STEP 1793 H=1793:00 +CPU= 21.472
  05:27:32 STEP 1794 H=1794:00 +CPU= 34.307
  05:27:54 STEP 1795 H=1795:00 +CPU= 21.522
  05:28:16 STEP 1796 H=1796:00 +CPU= 22.576
  05:28:51 STEP 1797 H=1797:00 +CPU= 34.844
  05:29:14 STEP 1798 H=1798:00 +CPU= 22.518
  05:29:35 STEP 1799 H=1799:00 +CPU= 21.435
  05:30:14 STEP 1800 H=1800:00 +CPU= 38.756
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001800
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001800
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001788
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001788
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001788
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_74.zip
Uploading the intermediate file: upload_file_74.zip
  05:30:35 STEP 1801 H=1801:00 +CPU= 20.371
  05:30:56 STEP 1802 H=1802:00 +CPU= 21.109
  05:31:30 STEP 1803 H=1803:00 +CPU= 34.356
  05:31:52 STEP 1804 H=1804:00 +CPU= 22.256
  05:32:16 STEP 1805 H=1805:00 +CPU= 23.964
  05:32:55 STEP 1806 H=1806:00 +CPU= 38.699
  05:33:19 STEP 1807 H=1807:00 +CPU= 23.790
  05:33:43 STEP 1808 H=1808:00 +CPU= 24.446
  05:34:20 STEP 1809 H=1809:00 +CPU= 36.271
  05:34:45 STEP 1810 H=1810:00 +CPU= 24.605
  05:35:08 STEP 1811 H=1811:00 +CPU= 23.804
  05:35:51 STEP 1812 H=1812:00 +CPU= 42.108
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001812
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001812
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001812
  05:36:15 STEP 1813 H=1813:00 +CPU= 23.822
  05:36:39 STEP 1814 H=1814:00 +CPU= 24.343
  05:37:17 STEP 1815 H=1815:00 +CPU= 37.588
  05:37:41 STEP 1816 H=1816:00 +CPU= 24.652
  05:38:05 STEP 1817 H=1817:00 +CPU= 23.562
  05:38:44 STEP 1818 H=1818:00 +CPU= 38.443
  05:39:07 STEP 1819 H=1819:00 +CPU= 23.277
  05:39:31 STEP 1820 H=1820:00 +CPU= 24.380
  05:40:09 STEP 1821 H=1821:00 +CPU= 37.337
  05:40:34 STEP 1822 H=1822:00 +CPU= 24.945
  05:40:59 STEP 1823 H=1823:00 +CPU= 25.413
  05:41:45 STEP 1824 H=1824:00 +CPU= 45.050
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001824
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001824
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001812
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001812
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001812
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_75.zip
Uploading the intermediate file: upload_file_75.zip
  05:42:08 STEP 1825 H=1825:00 +CPU= 23.736
  05:42:33 STEP 1826 H=1826:00 +CPU= 24.696
  05:43:11 STEP 1827 H=1827:00 +CPU= 38.046
  05:43:36 STEP 1828 H=1828:00 +CPU= 24.684
  05:44:01 STEP 1829 H=1829:00 +CPU= 24.825
  05:44:41 STEP 1830 H=1830:00 +CPU= 39.749
  05:45:04 STEP 1831 H=1831:00 +CPU= 22.864
  05:45:28 STEP 1832 H=1832:00 +CPU= 23.572
  05:46:03 STEP 1833 H=1833:00 +CPU= 35.678
  05:46:27 STEP 1834 H=1834:00 +CPU= 23.470
  05:46:50 STEP 1835 H=1835:00 +CPU= 22.780
  05:47:30 STEP 1836 H=1836:00 +CPU= 39.931
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001836
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001836
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001836
  05:47:53 STEP 1837 H=1837:00 +CPU= 22.751
  05:48:16 STEP 1838 H=1838:00 +CPU= 23.554
  05:48:52 STEP 1839 H=1839:00 +CPU= 35.979
  05:49:18 STEP 1840 H=1840:00 +CPU= 25.622
  05:49:43 STEP 1841 H=1841:00 +CPU= 24.927
  05:50:23 STEP 1842 H=1842:00 +CPU= 40.132
  05:50:48 STEP 1843 H=1843:00 +CPU= 24.711
  05:51:14 STEP 1844 H=1844:00 +CPU= 25.670
  05:51:53 STEP 1845 H=1845:00 +CPU= 39.221
  05:52:19 STEP 1846 H=1846:00 +CPU= 25.492
  05:52:43 STEP 1847 H=1847:00 +CPU= 24.513
  05:53:28 STEP 1848 H=1848:00 +CPU= 44.727
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001848
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001848
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001836
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001836
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001836
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_76.zip
Uploading the intermediate file: upload_file_76.zip
  05:53:52 STEP 1849 H=1849:00 +CPU= 23.412
Uploading trickle at timestep: 6652800
  05:54:16 STEP 1850 H=1850:00 +CPU= 24.369
  05:54:52 STEP 1851 H=1851:00 +CPU= 35.800
  05:55:16 STEP 1852 H=1852:00 +CPU= 23.546
  05:55:38 STEP 1853 H=1853:00 +CPU= 22.808
  05:56:15 STEP 1854 H=1854:00 +CPU= 36.726
  05:56:38 STEP 1855 H=1855:00 +CPU= 22.805
  05:57:02 STEP 1856 H=1856:00 +CPU= 23.694
  05:57:38 STEP 1857 H=1857:00 +CPU= 35.814
  05:58:01 STEP 1858 H=1858:00 +CPU= 23.547
  05:58:24 STEP 1859 H=1859:00 +CPU= 22.868
  05:59:04 STEP 1860 H=1860:00 +CPU= 40.001
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001860
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001860
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001860
  05:59:27 STEP 1861 H=1861:00 +CPU= 22.857
  05:59:51 STEP 1862 H=1862:00 +CPU= 23.656
  06:00:27 STEP 1863 H=1863:00 +CPU= 36.294
  06:00:52 STEP 1864 H=1864:00 +CPU= 24.454
  06:01:15 STEP 1865 H=1865:00 +CPU= 23.495
  06:01:54 STEP 1866 H=1866:00 +CPU= 38.602
  06:02:17 STEP 1867 H=1867:00 +CPU= 23.609
  06:02:42 STEP 1868 H=1868:00 +CPU= 24.681
  06:03:20 STEP 1869 H=1869:00 +CPU= 37.559
  06:03:44 STEP 1870 H=1870:00 +CPU= 24.441
  06:04:09 STEP 1871 H=1871:00 +CPU= 24.448
  06:04:54 STEP 1872 H=1872:00 +CPU= 44.617
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001872
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001872
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001860
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001860
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001860
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_77.zip
Uploading the intermediate file: upload_file_77.zip
  06:05:18 STEP 1873 H=1873:00 +CPU= 24.532
  06:05:44 STEP 1874 H=1874:00 +CPU= 25.491
  06:06:23 STEP 1875 H=1875:00 +CPU= 39.401
  06:06:50 STEP 1876 H=1876:00 +CPU= 26.285
  06:07:15 STEP 1877 H=1877:00 +CPU= 25.454
  06:07:55 STEP 1878 H=1878:00 +CPU= 39.918
  06:08:20 STEP 1879 H=1879:00 +CPU= 24.807
  06:08:46 STEP 1880 H=1880:00 +CPU= 25.605
  06:09:23 STEP 1881 H=1881:00 +CPU= 36.894
  06:09:47 STEP 1882 H=1882:00 +CPU= 23.782
  06:10:10 STEP 1883 H=1883:00 +CPU= 23.051
  06:10:50 STEP 1884 H=1884:00 +CPU= 40.378
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001884
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001884
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001884
  06:11:13 STEP 1885 H=1885:00 +CPU= 23.003
  06:11:37 STEP 1886 H=1886:00 +CPU= 23.781
  06:12:13 STEP 1887 H=1887:00 +CPU= 36.040
  06:12:37 STEP 1888 H=1888:00 +CPU= 23.784
  06:13:00 STEP 1889 H=1889:00 +CPU= 22.893
  06:13:37 STEP 1890 H=1890:00 +CPU= 37.390
  06:14:01 STEP 1891 H=1891:00 +CPU= 23.410
  06:14:25 STEP 1892 H=1892:00 +CPU= 24.130
  06:15:01 STEP 1893 H=1893:00 +CPU= 36.174
  06:15:25 STEP 1894 H=1894:00 +CPU= 23.788
  06:15:48 STEP 1895 H=1895:00 +CPU= 22.806
  06:16:32 STEP 1896 H=1896:00 +CPU= 43.942
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001896
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001896
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001884
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001884
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001884
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_78.zip
Uploading the intermediate file: upload_file_78.zip
  06:16:56 STEP 1897 H=1897:00 +CPU= 23.865
  06:17:21 STEP 1898 H=1898:00 +CPU= 24.720
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0104
(argv4) batchid: 981
(argv5) wuid: 12197748
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mydisks/a/boinc_lib/slots/14
Project directory is: /mydisks/a/boinc_lib/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748
..mkdir for temp folder for results failed
Copying: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c31547488b85abd7dee5be9fa77b1afd to: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_0104_2012050100_123_981_12197748.zip
Unzipping the namelist zip file: /mydisks/a/boinc_lib/slots/14/oifs_43r3_ps_0104_2012050100_123_981_12197748.zip
ic_ancil_file: ic_ancil_12197748
ifsdata_file: ifsdata_12197748
climate_data_file: clim_data_12197748
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f761aeb0b9f4f6620269cb483a6118dc to: /mydisks/a/boinc_lib/slots/14/ic_ancil_12197748.zip
Unzipping the IC ancils zip file: /mydisks/a/boinc_lib/slots/14/ic_ancil_12197748.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ec0332c27bf4e9a6bb5decbc16e70db6 to: /mydisks/a/boinc_lib/slots/14/ifsdata/ifsdata_12197748.zip
Unzipping the ifsdata_zip file: /mydisks/a/boinc_lib/slots/14/ifsdata/ifsdata_12197748.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/a/boinc_lib/slots/14/159l_2/clim_data_12197748.zip
Unzipping the climate data zip file: /mydisks/a/boinc_lib/slots/14/159l_2/clim_data_12197748.zip
Checking for progress XML file: /mydisks/a/boinc_lib/slots/14/progress_file_12197748.xml
Opened progress file ok : /mydisks/a/boinc_lib/slots/14/progress_file_12197748.xml
-- Model is restarting --
Adjusting last_iter, 1898, to previous model restart step.
Creating progress file: /mydisks/a/boinc_lib/slots/14/progress_file_12197748.xml
last_cpu_time: 57742
upload_file_number: 79
last_iter: 1895
last_upload: 6825600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431
The child process has been launched with process id: 917
Executing the command: /mydisks/a/boinc_lib/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:917:917] [20230125:062147:1674656507:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:22:30 STEP 1896 H=1896:00 +CPU= 40.736
  06:22:52 STEP 1897 H=1897:00 +CPU= 21.301
  06:23:13 STEP 1898 H=1898:00 +CPU= 21.751
  06:23:46 STEP 1899 H=1899:00 +CPU= 33.018
  06:24:08 STEP 1900 H=1900:00 +CPU= 21.923
  06:24:29 STEP 1901 H=1901:00 +CPU= 20.760
  06:25:03 STEP 1902 H=1902:00 +CPU= 34.075
  06:25:25 STEP 1903 H=1903:00 +CPU= 21.816
  06:25:47 STEP 1904 H=1904:00 +CPU= 22.020
  06:26:22 STEP 1905 H=1905:00 +CPU= 34.829
  06:26:44 STEP 1906 H=1906:00 +CPU= 22.127
  06:27:07 STEP 1907 H=1907:00 +CPU= 22.479
  06:27:47 STEP 1908 H=1908:00 +CPU= 40.190
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001908
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001908
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001908
  06:28:08 STEP 1909 H=1909:00 +CPU= 21.307
  06:28:30 STEP 1910 H=1910:00 +CPU= 21.938
  06:29:05 STEP 1911 H=1911:00 +CPU= 35.042
  06:29:27 STEP 1912 H=1912:00 +CPU= 22.122
  06:29:49 STEP 1913 H=1913:00 +CPU= 21.327
  06:30:24 STEP 1914 H=1914:00 +CPU= 35.323
  06:30:45 STEP 1915 H=1915:00 +CPU= 21.075
  06:31:07 STEP 1916 H=1916:00 +CPU= 21.611
  06:31:41 STEP 1917 H=1917:00 +CPU= 33.596
  06:32:02 STEP 1918 H=1918:00 +CPU= 21.576
  06:32:24 STEP 1919 H=1919:00 +CPU= 22.015
  06:33:05 STEP 1920 H=1920:00 +CPU= 40.392
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMGGhq0f+001920
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMSHhq0f+001920
Moving to projects directory: /mydisks/a/boinc_lib/slots/14/ICMUAhq0f+001920
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMGGhq0f+001908
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMSHhq0f+001908
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12197748/ICMUAhq0f+001908
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0104_2012050100_123_981_12197748_2_r1938594431_79.zip
Uploading the intermediate file: upload_file_79.zip
  06:33:26 STEP 1921 H=1921:00 +CPU= 21.113
  06:33:48 STEP 1922 H=1922:00 +CPU= 22.139
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:40:50 1517479 22302868 oifs_43r3_ps_0104_2012050100_123_981_12197748_2 6,652,800 56,326 0.0085
09 May 2023 16:40:24 1517479 22302868 oifs_43r3_ps_0104_2012050100_123_981_12197748_2 4,233,600 36,225 0.0086
09 May 2023 16:40:14 1517479 22302868 oifs_43r3_ps_0104_2012050100_123_981_12197748_2 2,592,000 22,815 0.0088
09 May 2023 16:39:31 1517479 22302868 oifs_43r3_ps_0104_2012050100_123_981_12197748_2 1,728,000 15,153 0.0088
09 May 2023 16:39:23 1517479 22302868 oifs_43r3_ps_0104_2012050100_123_981_12197748_2 864,000 7,645 0.0088
09 May 2023 16:39:23 1517479 22302868 oifs_43r3_ps_0104_2012050100_123_981_12197748_2 864,000 7,645 0.0088


©2024 climateprediction.net