climateprediction.net home page
Task 22302847

Task 22302847

Name oifs_43r3_ps_0769_2015050100_123_984_12201413_1
Workunit 12201413
Created 24 Jan 2023, 9:58:25 UTC
Sent 24 Jan 2023, 10:36:40 UTC
Report deadline 25 Mar 2023, 10:36:40 UTC
Received 25 Jan 2023, 15:19:15 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1517479
Run time 16 hours 25 min 53 sec
CPU time 16 hours 21 min 35 sec
Validate state Invalid
Credit 9,313.92
Device peak FLOPS 4.36 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.79 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,989.61 MB
Stderr
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
a/boinc_lib/slots/30/ICMGGhq0f+001488
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001488
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001464
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001464
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001464
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001476
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001476
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001476
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_61.zip
Uploading the intermediate file: upload_file_61.zip
  02:53:15 STEP 1489 H=1489:00 +CPU= 21.303
  02:53:38 STEP 1490 H=1490:00 +CPU= 22.889
  02:54:11 STEP 1491 H=1491:00 +CPU= 33.320
  02:54:35 STEP 1492 H=1492:00 +CPU= 23.365
  02:54:56 STEP 1493 H=1493:00 +CPU= 20.959
  02:55:29 STEP 1494 H=1494:00 +CPU= 33.333
  02:55:50 STEP 1495 H=1495:00 +CPU= 20.378
  02:56:11 STEP 1496 H=1496:00 +CPU= 20.992
  02:56:44 STEP 1497 H=1497:00 +CPU= 33.163
  02:57:05 STEP 1498 H=1498:00 +CPU= 21.454
  02:57:26 STEP 1499 H=1499:00 +CPU= 20.194
  02:58:02 STEP 1500 H=1500:00 +CPU= 36.779
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001500
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001500
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001500
  02:58:23 STEP 1501 H=1501:00 +CPU= 20.483
  02:58:45 STEP 1502 H=1502:00 +CPU= 21.998
  02:59:18 STEP 1503 H=1503:00 +CPU= 33.332
  02:59:40 STEP 1504 H=1504:00 +CPU= 21.708
  03:00:01 STEP 1505 H=1505:00 +CPU= 21.122
  03:00:35 STEP 1506 H=1506:00 +CPU= 34.110
  03:00:56 STEP 1507 H=1507:00 +CPU= 20.734
  03:01:17 STEP 1508 H=1508:00 +CPU= 21.229
  03:01:50 STEP 1509 H=1509:00 +CPU= 32.686
  03:02:11 STEP 1510 H=1510:00 +CPU= 21.328
  03:02:33 STEP 1511 H=1511:00 +CPU= 21.387
  03:03:12 STEP 1512 H=1512:00 +CPU= 39.224
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001512
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001512
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001488
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001500
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001500
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001500
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_62.zip
Uploading the intermediate file: upload_file_62.zip
  03:03:33 STEP 1513 H=1513:00 +CPU= 20.734
  03:03:55 STEP 1514 H=1514:00 +CPU= 21.940
  03:04:29 STEP 1515 H=1515:00 +CPU= 33.693
  03:04:51 STEP 1516 H=1516:00 +CPU= 21.995
  03:05:12 STEP 1517 H=1517:00 +CPU= 21.511
  03:05:47 STEP 1518 H=1518:00 +CPU= 34.474
  03:06:08 STEP 1519 H=1519:00 +CPU= 21.053
  03:06:29 STEP 1520 H=1520:00 +CPU= 21.710
  03:07:03 STEP 1521 H=1521:00 +CPU= 33.512
  03:07:24 STEP 1522 H=1522:00 +CPU= 21.273
  03:07:45 STEP 1523 H=1523:00 +CPU= 20.348
  03:08:21 STEP 1524 H=1524:00 +CPU= 36.008
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001524
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001524
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001524
  03:08:42 STEP 1525 H=1525:00 +CPU= 20.842
  03:09:04 STEP 1526 H=1526:00 +CPU= 22.004
  03:09:37 STEP 1527 H=1527:00 +CPU= 33.459
  03:09:59 STEP 1528 H=1528:00 +CPU= 21.660
  03:10:20 STEP 1529 H=1529:00 +CPU= 20.784
  03:10:54 STEP 1530 H=1530:00 +CPU= 33.929
  03:11:15 STEP 1531 H=1531:00 +CPU= 20.911
  03:11:36 STEP 1532 H=1532:00 +CPU= 21.887
  03:12:10 STEP 1533 H=1533:00 +CPU= 33.783
  03:12:32 STEP 1534 H=1534:00 +CPU= 21.890
  03:12:54 STEP 1535 H=1535:00 +CPU= 21.304
  03:13:37 STEP 1536 H=1536:00 +CPU= 43.298
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001536
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001536
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001512
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001524
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001524
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001524
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_63.zip
Uploading the intermediate file: upload_file_63.zip
  03:13:59 STEP 1537 H=1537:00 +CPU= 21.587
  03:14:20 STEP 1538 H=1538:00 +CPU= 21.960
  03:14:56 STEP 1539 H=1539:00 +CPU= 35.533
  03:15:18 STEP 1540 H=1540:00 +CPU= 21.869
  03:15:41 STEP 1541 H=1541:00 +CPU= 22.700
  03:16:15 STEP 1542 H=1542:00 +CPU= 34.524
  03:16:36 STEP 1543 H=1543:00 +CPU= 20.806
  03:16:58 STEP 1544 H=1544:00 +CPU= 21.638
  03:17:34 STEP 1545 H=1545:00 +CPU= 35.899
  03:17:55 STEP 1546 H=1546:00 +CPU= 21.439
  03:18:16 STEP 1547 H=1547:00 +CPU= 20.710
  03:18:53 STEP 1548 H=1548:00 +CPU= 36.491
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001548
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001548
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001548
  03:19:13 STEP 1549 H=1549:00 +CPU= 20.480
  03:19:35 STEP 1550 H=1550:00 +CPU= 22.427
  03:20:10 STEP 1551 H=1551:00 +CPU= 34.068
  03:20:31 STEP 1552 H=1552:00 +CPU= 21.140
  03:20:53 STEP 1553 H=1553:00 +CPU= 22.284
  03:21:27 STEP 1554 H=1554:00 +CPU= 34.021
  03:21:51 STEP 1555 H=1555:00 +CPU= 23.530
  03:22:16 STEP 1556 H=1556:00 +CPU= 25.542
  03:22:56 STEP 1557 H=1557:00 +CPU= 39.527
  03:23:22 STEP 1558 H=1558:00 +CPU= 25.990
  03:23:46 STEP 1559 H=1559:00 +CPU= 23.954
  03:24:29 STEP 1560 H=1560:00 +CPU= 42.607
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001560
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001560
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001536
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001548
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001548
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001548
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_64.zip
Uploading the intermediate file: upload_file_64.zip
  03:24:52 STEP 1561 H=1561:00 +CPU= 22.973
  03:25:16 STEP 1562 H=1562:00 +CPU= 23.802
  03:25:52 STEP 1563 H=1563:00 +CPU= 35.943
  03:26:16 STEP 1564 H=1564:00 +CPU= 23.754
  03:26:39 STEP 1565 H=1565:00 +CPU= 22.998
  03:27:16 STEP 1566 H=1566:00 +CPU= 36.948
  03:27:39 STEP 1567 H=1567:00 +CPU= 23.029
  03:28:03 STEP 1568 H=1568:00 +CPU= 23.798
  03:28:39 STEP 1569 H=1569:00 +CPU= 36.033
  03:29:03 STEP 1570 H=1570:00 +CPU= 24.110
  03:29:26 STEP 1571 H=1571:00 +CPU= 23.426
  03:30:07 STEP 1572 H=1572:00 +CPU= 41.036
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001572
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001572
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001572
  03:30:30 STEP 1573 H=1573:00 +CPU= 23.110
  03:30:54 STEP 1574 H=1574:00 +CPU= 23.761
  03:31:31 STEP 1575 H=1575:00 +CPU= 36.108
  03:31:54 STEP 1576 H=1576:00 +CPU= 23.933
  03:32:18 STEP 1577 H=1577:00 +CPU= 23.044
  03:32:55 STEP 1578 H=1578:00 +CPU= 36.911
  03:33:18 STEP 1579 H=1579:00 +CPU= 22.956
  03:33:41 STEP 1580 H=1580:00 +CPU= 23.849
  03:34:19 STEP 1581 H=1581:00 +CPU= 37.551
  03:34:43 STEP 1582 H=1582:00 +CPU= 24.297
  03:35:08 STEP 1583 H=1583:00 +CPU= 24.469
  03:35:53 STEP 1584 H=1584:00 +CPU= 44.581
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001584
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001584
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001560
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001572
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001572
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001572
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_65.zip
Uploading the intermediate file: upload_file_65.zip
  03:36:17 STEP 1585 H=1585:00 +CPU= 23.629
  03:36:41 STEP 1586 H=1586:00 +CPU= 24.388
  03:37:19 STEP 1587 H=1587:00 +CPU= 37.708
  03:37:44 STEP 1588 H=1588:00 +CPU= 24.716
  03:38:08 STEP 1589 H=1589:00 +CPU= 23.819
  03:38:46 STEP 1590 H=1590:00 +CPU= 37.672
  03:39:09 STEP 1591 H=1591:00 +CPU= 23.279
  03:39:33 STEP 1592 H=1592:00 +CPU= 24.493
  03:40:11 STEP 1593 H=1593:00 +CPU= 37.324
  03:40:35 STEP 1594 H=1594:00 +CPU= 23.736
  03:41:00 STEP 1595 H=1595:00 +CPU= 24.631
  03:41:44 STEP 1596 H=1596:00 +CPU= 43.868
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001596
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001596
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001596
  03:42:09 STEP 1597 H=1597:00 +CPU= 25.174
  03:42:34 STEP 1598 H=1598:00 +CPU= 24.647
  03:43:11 STEP 1599 H=1599:00 +CPU= 36.992
  03:43:36 STEP 1600 H=1600:00 +CPU= 24.957
  03:44:01 STEP 1601 H=1601:00 +CPU= 24.100
  03:44:40 STEP 1602 H=1602:00 +CPU= 39.277
  03:45:05 STEP 1603 H=1603:00 +CPU= 24.205
  03:45:29 STEP 1604 H=1604:00 +CPU= 24.135
  03:46:06 STEP 1605 H=1605:00 +CPU= 36.459
  03:46:30 STEP 1606 H=1606:00 +CPU= 24.285
  03:46:55 STEP 1607 H=1607:00 +CPU= 24.674
  03:47:40 STEP 1608 H=1608:00 +CPU= 45.096
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001608
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001608
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001584
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001596
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001596
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001596
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_66.zip
Uploading the intermediate file: upload_file_66.zip
  03:48:04 STEP 1609 H=1609:00 +CPU= 24.069
  03:48:29 STEP 1610 H=1610:00 +CPU= 24.593
  03:49:07 STEP 1611 H=1611:00 +CPU= 37.983
  03:49:32 STEP 1612 H=1612:00 +CPU= 24.931
  03:49:56 STEP 1613 H=1613:00 +CPU= 23.978
  03:50:36 STEP 1614 H=1614:00 +CPU= 39.271
  03:51:00 STEP 1615 H=1615:00 +CPU= 24.122
  03:51:25 STEP 1616 H=1616:00 +CPU= 24.842
  03:52:03 STEP 1617 H=1617:00 +CPU= 38.309
  03:52:28 STEP 1618 H=1618:00 +CPU= 24.846
  03:52:52 STEP 1619 H=1619:00 +CPU= 24.002
  03:53:35 STEP 1620 H=1620:00 +CPU= 42.578
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001620
  03:53:59 STEP 1621 H=1621:00 +CPU= 23.995
  03:54:23 STEP 1622 H=1622:00 +CPU= 24.659
  03:55:02 STEP 1623 H=1623:00 +CPU= 38.003
  03:55:27 STEP 1624 H=1624:00 +CPU= 24.935
  03:55:50 STEP 1625 H=1625:00 +CPU= 23.757
  03:56:30 STEP 1626 H=1626:00 +CPU= 39.055
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0769
(argv4) batchid: 984
(argv5) wuid: 12201413
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mydisks/a/boinc_lib/slots/30
Project directory is: /mydisks/a/boinc_lib/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413
..mkdir for temp folder for results failed
Copying: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6ccc10347dc5194afae77b400e006e0d to: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_0769_2015050100_123_984_12201413.zip
Unzipping the namelist zip file: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_0769_2015050100_123_984_12201413.zip
ic_ancil_file: ic_ancil_12201413
ifsdata_file: ifsdata_12201413
climate_data_file: clim_data_12201413
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c3a7a1d21fa2cbee2d96b97baf803571 to: /mydisks/a/boinc_lib/slots/30/ic_ancil_12201413.zip
Unzipping the IC ancils zip file: /mydisks/a/boinc_lib/slots/30/ic_ancil_12201413.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c0a55765f2ed03ab43d4fbe2239c9369 to: /mydisks/a/boinc_lib/slots/30/ifsdata/ifsdata_12201413.zip
Unzipping the ifsdata_zip file: /mydisks/a/boinc_lib/slots/30/ifsdata/ifsdata_12201413.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/a/boinc_lib/slots/30/159l_2/clim_data_12201413.zip
Unzipping the climate data zip file: /mydisks/a/boinc_lib/slots/30/159l_2/clim_data_12201413.zip
Checking for progress XML file: /mydisks/a/boinc_lib/slots/30/progress_file_12201413.xml
Opened progress file ok : /mydisks/a/boinc_lib/slots/30/progress_file_12201413.xml
-- Model is restarting --
Adjusting last_iter, 1627, to previous model restart step.
Creating progress file: /mydisks/a/boinc_lib/slots/30/progress_file_12201413.xml
last_cpu_time: 50135
upload_file_number: 67
last_iter: 1607
last_upload: 5788800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781
The child process has been launched with process id: 8108
Executing the command: /mydisks/a/boinc_lib/slots/30/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:8108:8108] [20230125:040437:1674648277:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:05:20 STEP 1608 H=1608:00 +CPU= 40.209
  04:05:41 STEP 1609 H=1609:00 +CPU= 21.104
  04:06:04 STEP 1610 H=1610:00 +CPU= 23.267
  04:06:40 STEP 1611 H=1611:00 +CPU= 35.121
  04:07:01 STEP 1612 H=1612:00 +CPU= 21.723
  04:07:22 STEP 1613 H=1613:00 +CPU= 20.861
  04:07:56 STEP 1614 H=1614:00 +CPU= 33.777
  04:08:17 STEP 1615 H=1615:00 +CPU= 20.649
  04:08:38 STEP 1616 H=1616:00 +CPU= 21.503
  04:09:11 STEP 1617 H=1617:00 +CPU= 33.117
  04:09:33 STEP 1618 H=1618:00 +CPU= 21.396
  04:09:54 STEP 1619 H=1619:00 +CPU= 21.170
  04:10:31 STEP 1620 H=1620:00 +CPU= 37.315
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001620
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001620
  04:10:52 STEP 1621 H=1621:00 +CPU= 20.607
  04:11:14 STEP 1622 H=1622:00 +CPU= 21.856
  04:11:47 STEP 1623 H=1623:00 +CPU= 33.584
  04:12:09 STEP 1624 H=1624:00 +CPU= 21.887
  04:12:30 STEP 1625 H=1625:00 +CPU= 21.025
  04:13:05 STEP 1626 H=1626:00 +CPU= 34.220
  04:13:26 STEP 1627 H=1627:00 +CPU= 20.788
  04:13:47 STEP 1628 H=1628:00 +CPU= 21.142
  04:14:20 STEP 1629 H=1629:00 +CPU= 33.270
  04:14:41 STEP 1630 H=1630:00 +CPU= 21.353
  04:15:03 STEP 1631 H=1631:00 +CPU= 21.166
  04:15:44 STEP 1632 H=1632:00 +CPU= 41.401
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001632
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001632
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001608
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001620
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001620
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001620
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_67.zip
Uploading the intermediate file: upload_file_67.zip
  04:16:05 STEP 1633 H=1633:00 +CPU= 20.641
  04:16:27 STEP 1634 H=1634:00 +CPU= 21.764
  04:17:00 STEP 1635 H=1635:00 +CPU= 33.104
  04:17:21 STEP 1636 H=1636:00 +CPU= 21.288
  04:17:41 STEP 1637 H=1637:00 +CPU= 20.496
  04:18:15 STEP 1638 H=1638:00 +CPU= 33.780
  04:18:36 STEP 1639 H=1639:00 +CPU= 20.758
  04:18:58 STEP 1640 H=1640:00 +CPU= 21.878
  04:19:34 STEP 1641 H=1641:00 +CPU= 36.151
  04:19:57 STEP 1642 H=1642:00 +CPU= 23.067
  04:20:18 STEP 1643 H=1643:00 +CPU= 20.644
  04:20:56 STEP 1644 H=1644:00 +CPU= 37.535
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001644
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001644
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001644
  04:21:17 STEP 1645 H=1645:00 +CPU= 20.836
  04:21:38 STEP 1646 H=1646:00 +CPU= 21.163
  04:22:11 STEP 1647 H=1647:00 +CPU= 33.447
  04:22:32 STEP 1648 H=1648:00 +CPU= 21.089
  04:22:53 STEP 1649 H=1649:00 +CPU= 20.267
  04:23:26 STEP 1650 H=1650:00 +CPU= 33.101
  04:23:47 STEP 1651 H=1651:00 +CPU= 21.483
  04:24:09 STEP 1652 H=1652:00 +CPU= 21.508
  04:24:37 STEP 1653 H=1653:00 +CPU= 27.893
  04:24:55 STEP 1654 H=1654:00 +CPU= 17.852
  04:25:17 STEP 1655 H=1655:00 +CPU= 21.991
  04:25:58 STEP 1656 H=1656:00 +CPU= 40.841
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001656
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001656
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001632
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001644
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001644
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001644
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_68.zip
Uploading the intermediate file: upload_file_68.zip
  04:26:19 STEP 1657 H=1657:00 +CPU= 21.063
  04:26:43 STEP 1658 H=1658:00 +CPU= 23.913
  04:27:16 STEP 1659 H=1659:00 +CPU= 33.493
  04:27:38 STEP 1660 H=1660:00 +CPU= 21.462
  04:27:58 STEP 1661 H=1661:00 +CPU= 20.488
  04:28:32 STEP 1662 H=1662:00 +CPU= 33.378
  04:28:54 STEP 1663 H=1663:00 +CPU= 22.573
  04:29:18 STEP 1664 H=1664:00 +CPU= 23.402
  04:29:51 STEP 1665 H=1665:00 +CPU= 33.414
  04:30:12 STEP 1666 H=1666:00 +CPU= 21.056
  04:30:32 STEP 1667 H=1667:00 +CPU= 20.154
  04:31:09 STEP 1668 H=1668:00 +CPU= 36.977
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001668
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001668
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001668
  04:31:30 STEP 1669 H=1669:00 +CPU= 20.924
  04:31:52 STEP 1670 H=1670:00 +CPU= 21.551
  04:32:25 STEP 1671 H=1671:00 +CPU= 33.155
  04:32:47 STEP 1672 H=1672:00 +CPU= 21.648
  04:33:07 STEP 1673 H=1673:00 +CPU= 20.076
  04:33:37 STEP 1674 H=1674:00 +CPU= 29.733
  04:33:52 STEP 1675 H=1675:00 +CPU= 15.247
  04:34:08 STEP 1676 H=1676:00 +CPU= 15.734
  04:34:36 STEP 1677 H=1677:00 +CPU= 28.746
  04:34:58 STEP 1678 H=1678:00 +CPU= 21.803
  04:35:21 STEP 1679 H=1679:00 +CPU= 22.190
  04:36:01 STEP 1680 H=1680:00 +CPU= 40.704
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001680
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001680
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001656
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001668
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001668
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001668
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_69.zip
Uploading the intermediate file: upload_file_69.zip
  04:36:24 STEP 1681 H=1681:00 +CPU= 22.340
  04:36:47 STEP 1682 H=1682:00 +CPU= 23.545
  04:37:21 STEP 1683 H=1683:00 +CPU= 33.860
  04:37:43 STEP 1684 H=1684:00 +CPU= 21.543
  04:38:04 STEP 1685 H=1685:00 +CPU= 21.617
  04:38:40 STEP 1686 H=1686:00 +CPU= 35.758
  04:39:02 STEP 1687 H=1687:00 +CPU= 21.755
  04:39:28 STEP 1688 H=1688:00 +CPU= 26.024
  04:40:03 STEP 1689 H=1689:00 +CPU= 34.938
  04:40:25 STEP 1690 H=1690:00 +CPU= 21.668
  04:40:46 STEP 1691 H=1691:00 +CPU= 20.914
  04:41:23 STEP 1692 H=1692:00 +CPU= 37.711
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001692
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001692
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001692
  04:41:44 STEP 1693 H=1693:00 +CPU= 20.911
  04:42:07 STEP 1694 H=1694:00 +CPU= 22.424
  04:42:45 STEP 1695 H=1695:00 +CPU= 37.989
  04:43:08 STEP 1696 H=1696:00 +CPU= 23.110
  04:43:29 STEP 1697 H=1697:00 +CPU= 20.988
  04:44:04 STEP 1698 H=1698:00 +CPU= 35.210
  04:44:27 STEP 1699 H=1699:00 +CPU= 23.148
  04:44:50 STEP 1700 H=1700:00 +CPU= 22.727
  04:45:25 STEP 1701 H=1701:00 +CPU= 34.750
  04:45:48 STEP 1702 H=1702:00 +CPU= 22.891
  04:46:11 STEP 1703 H=1703:00 +CPU= 22.960
  04:46:52 STEP 1704 H=1704:00 +CPU= 40.709
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001704
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001704
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001680
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001692
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001692
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001692
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_70.zip
Uploading the intermediate file: upload_file_70.zip
  04:47:14 STEP 1705 H=1705:00 +CPU= 21.972
  04:47:38 STEP 1706 H=1706:00 +CPU= 23.774
  04:48:15 STEP 1707 H=1707:00 +CPU= 37.433
  04:48:38 STEP 1708 H=1708:00 +CPU= 22.597
  04:48:59 STEP 1709 H=1709:00 +CPU= 21.214
  04:49:34 STEP 1710 H=1710:00 +CPU= 34.803
  04:49:55 STEP 1711 H=1711:00 +CPU= 21.144
  04:50:17 STEP 1712 H=1712:00 +CPU= 21.876
  04:50:51 STEP 1713 H=1713:00 +CPU= 34.001
  04:51:14 STEP 1714 H=1714:00 +CPU= 22.886
  04:51:35 STEP 1715 H=1715:00 +CPU= 20.979
  04:52:13 STEP 1716 H=1716:00 +CPU= 37.550
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001716
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001716
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001716
  04:52:35 STEP 1717 H=1717:00 +CPU= 22.021
  04:53:01 STEP 1718 H=1718:00 +CPU= 25.998
  04:53:37 STEP 1719 H=1719:00 +CPU= 36.862
  04:54:00 STEP 1720 H=1720:00 +CPU= 22.066
  04:54:21 STEP 1721 H=1721:00 +CPU= 21.378
  04:54:57 STEP 1722 H=1722:00 +CPU= 36.192
  04:55:20 STEP 1723 H=1723:00 +CPU= 22.271
  04:55:42 STEP 1724 H=1724:00 +CPU= 22.725
  04:56:17 STEP 1725 H=1725:00 +CPU= 34.494
  04:56:40 STEP 1726 H=1726:00 +CPU= 23.024
  04:57:02 STEP 1727 H=1727:00 +CPU= 21.720
  04:57:42 STEP 1728 H=1728:00 +CPU= 40.117
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001728
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001728
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001704
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001716
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001716
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001716
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_71.zip
Uploading the intermediate file: upload_file_71.zip
  04:58:03 STEP 1729 H=1729:00 +CPU= 21.284
  04:58:25 STEP 1730 H=1730:00 +CPU= 22.102
  04:59:02 STEP 1731 H=1731:00 +CPU= 36.572
  04:59:26 STEP 1732 H=1732:00 +CPU= 23.967
  04:59:48 STEP 1733 H=1733:00 +CPU= 21.959
  05:00:25 STEP 1734 H=1734:00 +CPU= 37.019
  05:00:47 STEP 1735 H=1735:00 +CPU= 21.608
  05:01:09 STEP 1736 H=1736:00 +CPU= 21.985
  05:01:43 STEP 1737 H=1737:00 +CPU= 34.280
  05:02:05 STEP 1738 H=1738:00 +CPU= 21.899
  05:02:26 STEP 1739 H=1739:00 +CPU= 20.917
  05:03:05 STEP 1740 H=1740:00 +CPU= 38.762
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001740
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001740
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001740
  05:03:26 STEP 1741 H=1741:00 +CPU= 21.608
  05:03:49 STEP 1742 H=1742:00 +CPU= 22.191
  05:04:22 STEP 1743 H=1743:00 +CPU= 33.851
  05:04:46 STEP 1744 H=1744:00 +CPU= 23.253
  05:05:08 STEP 1745 H=1745:00 +CPU= 22.514
  05:05:44 STEP 1746 H=1746:00 +CPU= 35.436
  05:06:06 STEP 1747 H=1747:00 +CPU= 21.929
  05:06:29 STEP 1748 H=1748:00 +CPU= 22.751
  05:07:03 STEP 1749 H=1749:00 +CPU= 34.774
  05:07:26 STEP 1750 H=1750:00 +CPU= 22.763
  05:07:49 STEP 1751 H=1751:00 +CPU= 22.382
  05:08:29 STEP 1752 H=1752:00 +CPU= 40.529
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001752
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001752
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001728
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001740
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001740
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001740
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_72.zip
Uploading the intermediate file: upload_file_72.zip
  05:08:51 STEP 1753 H=1753:00 +CPU= 21.797
  05:09:14 STEP 1754 H=1754:00 +CPU= 22.339
  05:09:48 STEP 1755 H=1755:00 +CPU= 34.019
  05:10:10 STEP 1756 H=1756:00 +CPU= 22.371
  05:10:32 STEP 1757 H=1757:00 +CPU= 21.586
  05:11:07 STEP 1758 H=1758:00 +CPU= 34.974
  05:11:29 STEP 1759 H=1759:00 +CPU= 22.571
  05:11:51 STEP 1760 H=1760:00 +CPU= 22.041
  05:12:25 STEP 1761 H=1761:00 +CPU= 33.836
  05:12:48 STEP 1762 H=1762:00 +CPU= 22.462
  05:13:11 STEP 1763 H=1763:00 +CPU= 22.823
  05:13:49 STEP 1764 H=1764:00 +CPU= 38.732
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001764
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001764
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001764
  05:14:11 STEP 1765 H=1765:00 +CPU= 22.011
  05:14:34 STEP 1766 H=1766:00 +CPU= 22.875
  05:15:08 STEP 1767 H=1767:00 +CPU= 33.727
  05:15:31 STEP 1768 H=1768:00 +CPU= 22.612
  05:15:52 STEP 1769 H=1769:00 +CPU= 21.202
  05:16:27 STEP 1770 H=1770:00 +CPU= 35.091
  05:16:50 STEP 1771 H=1771:00 +CPU= 22.659
  05:17:12 STEP 1772 H=1772:00 +CPU= 22.419
  05:17:47 STEP 1773 H=1773:00 +CPU= 34.256
  05:18:10 STEP 1774 H=1774:00 +CPU= 23.355
  05:18:33 STEP 1775 H=1775:00 +CPU= 22.962
  05:19:14 STEP 1776 H=1776:00 +CPU= 41.300
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001776
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001776
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001752
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001764
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001764
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001764
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_73.zip
Uploading the intermediate file: upload_file_73.zip
  05:19:36 STEP 1777 H=1777:00 +CPU= 21.442
  05:19:58 STEP 1778 H=1778:00 +CPU= 22.615
  05:20:34 STEP 1779 H=1779:00 +CPU= 35.064
  05:20:56 STEP 1780 H=1780:00 +CPU= 22.597
  05:21:18 STEP 1781 H=1781:00 +CPU= 22.036
  05:21:54 STEP 1782 H=1782:00 +CPU= 35.858
  05:22:16 STEP 1783 H=1783:00 +CPU= 21.376
  05:22:38 STEP 1784 H=1784:00 +CPU= 22.071
  05:23:12 STEP 1785 H=1785:00 +CPU= 34.590
  05:23:35 STEP 1786 H=1786:00 +CPU= 22.423
  05:23:56 STEP 1787 H=1787:00 +CPU= 21.350
  05:24:34 STEP 1788 H=1788:00 +CPU= 37.904
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001788
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001788
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001788
  05:24:55 STEP 1789 H=1789:00 +CPU= 21.313
  05:25:17 STEP 1790 H=1790:00 +CPU= 21.704
  05:25:52 STEP 1791 H=1791:00 +CPU= 35.083
  05:26:17 STEP 1792 H=1792:00 +CPU= 25.184
  05:26:40 STEP 1793 H=1793:00 +CPU= 22.232
  05:27:14 STEP 1794 H=1794:00 +CPU= 34.540
  05:27:35 STEP 1795 H=1795:00 +CPU= 20.651
  05:27:57 STEP 1796 H=1796:00 +CPU= 22.339
  05:28:32 STEP 1797 H=1797:00 +CPU= 34.327
  05:28:56 STEP 1798 H=1798:00 +CPU= 24.417
  05:29:20 STEP 1799 H=1799:00 +CPU= 23.269
  05:29:58 STEP 1800 H=1800:00 +CPU= 38.722
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001800
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001800
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001776
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001788
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001788
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001788
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_74.zip
Uploading the intermediate file: upload_file_74.zip
  05:30:18 STEP 1801 H=1801:00 +CPU= 20.155
  05:30:40 STEP 1802 H=1802:00 +CPU= 21.078
  05:31:13 STEP 1803 H=1803:00 +CPU= 33.182
  05:31:34 STEP 1804 H=1804:00 +CPU= 21.128
  05:31:54 STEP 1805 H=1805:00 +CPU= 20.135
  05:32:33 STEP 1806 H=1806:00 +CPU= 38.531
  05:32:56 STEP 1807 H=1807:00 +CPU= 23.343
  05:33:20 STEP 1808 H=1808:00 +CPU= 23.808
  05:33:56 STEP 1809 H=1809:00 +CPU= 35.590
  05:34:19 STEP 1810 H=1810:00 +CPU= 23.649
  05:34:42 STEP 1811 H=1811:00 +CPU= 22.920
  05:35:22 STEP 1812 H=1812:00 +CPU= 39.769
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001812
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001812
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001812
  05:35:45 STEP 1813 H=1813:00 +CPU= 22.767
  05:36:09 STEP 1814 H=1814:00 +CPU= 23.567
  05:36:44 STEP 1815 H=1815:00 +CPU= 35.637
  05:37:08 STEP 1816 H=1816:00 +CPU= 23.572
  05:37:31 STEP 1817 H=1817:00 +CPU= 22.745
  05:38:07 STEP 1818 H=1818:00 +CPU= 36.451
  05:38:30 STEP 1819 H=1819:00 +CPU= 22.839
  05:38:55 STEP 1820 H=1820:00 +CPU= 24.155
  05:39:30 STEP 1821 H=1821:00 +CPU= 35.769
  05:39:54 STEP 1822 H=1822:00 +CPU= 23.454
  05:40:18 STEP 1823 H=1823:00 +CPU= 24.220
  05:41:05 STEP 1824 H=1824:00 +CPU= 46.381
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001824
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001824
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001800
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001812
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001812
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001812
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_75.zip
Uploading the intermediate file: upload_file_75.zip
  05:41:28 STEP 1825 H=1825:00 +CPU= 23.384
  05:41:52 STEP 1826 H=1826:00 +CPU= 23.410
  05:42:29 STEP 1827 H=1827:00 +CPU= 37.335
  05:42:53 STEP 1828 H=1828:00 +CPU= 24.479
  05:43:17 STEP 1829 H=1829:00 +CPU= 23.732
  05:43:56 STEP 1830 H=1830:00 +CPU= 38.619
  05:44:20 STEP 1831 H=1831:00 +CPU= 23.711
  05:44:44 STEP 1832 H=1832:00 +CPU= 24.556
  05:45:21 STEP 1833 H=1833:00 +CPU= 37.180
  05:45:45 STEP 1834 H=1834:00 +CPU= 23.998
  05:46:09 STEP 1835 H=1835:00 +CPU= 23.482
  05:46:51 STEP 1836 H=1836:00 +CPU= 42.067
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001836
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001836
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001836
  05:47:15 STEP 1837 H=1837:00 +CPU= 23.611
  05:47:39 STEP 1838 H=1838:00 +CPU= 24.296
  05:48:16 STEP 1839 H=1839:00 +CPU= 36.937
  05:48:41 STEP 1840 H=1840:00 +CPU= 24.484
  05:49:04 STEP 1841 H=1841:00 +CPU= 23.679
  05:49:42 STEP 1842 H=1842:00 +CPU= 38.067
  05:50:06 STEP 1843 H=1843:00 +CPU= 23.376
  05:50:32 STEP 1844 H=1844:00 +CPU= 25.612
  05:51:11 STEP 1845 H=1845:00 +CPU= 39.065
  05:51:36 STEP 1846 H=1846:00 +CPU= 25.494
  05:52:01 STEP 1847 H=1847:00 +CPU= 24.409
  05:52:45 STEP 1848 H=1848:00 +CPU= 43.809
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001848
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001848
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001824
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001836
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001836
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001836
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_76.zip
Uploading the intermediate file: upload_file_76.zip
Uploading trickle at timestep: 6652800
  05:53:08 STEP 1849 H=1849:00 +CPU= 23.216
  05:53:33 STEP 1850 H=1850:00 +CPU= 24.421
  05:54:08 STEP 1851 H=1851:00 +CPU= 35.385
  05:54:32 STEP 1852 H=1852:00 +CPU= 23.482
  05:54:55 STEP 1853 H=1853:00 +CPU= 22.807
  05:55:31 STEP 1854 H=1854:00 +CPU= 36.694
  05:55:54 STEP 1855 H=1855:00 +CPU= 22.839
  05:56:18 STEP 1856 H=1856:00 +CPU= 23.682
  05:56:54 STEP 1857 H=1857:00 +CPU= 35.798
  05:57:18 STEP 1858 H=1858:00 +CPU= 23.649
  05:57:40 STEP 1859 H=1859:00 +CPU= 22.943
  05:58:21 STEP 1860 H=1860:00 +CPU= 40.576
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001860
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001860
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001860
  05:58:45 STEP 1861 H=1861:00 +CPU= 23.714
  05:59:10 STEP 1862 H=1862:00 +CPU= 24.602
  05:59:48 STEP 1863 H=1863:00 +CPU= 37.580
  06:00:13 STEP 1864 H=1864:00 +CPU= 24.120
  06:00:37 STEP 1865 H=1865:00 +CPU= 23.459
  06:01:17 STEP 1866 H=1866:00 +CPU= 39.940
  06:01:40 STEP 1867 H=1867:00 +CPU= 23.719
  06:02:05 STEP 1868 H=1868:00 +CPU= 24.294
  06:02:43 STEP 1869 H=1869:00 +CPU= 38.046
  06:03:08 STEP 1870 H=1870:00 +CPU= 25.509
  06:03:34 STEP 1871 H=1871:00 +CPU= 25.658
  06:04:21 STEP 1872 H=1872:00 +CPU= 46.227
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001872
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001872
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001848
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001860
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001860
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001860
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_77.zip
Uploading the intermediate file: upload_file_77.zip
  06:04:45 STEP 1873 H=1873:00 +CPU= 24.749
  06:05:11 STEP 1874 H=1874:00 +CPU= 25.654
  06:05:50 STEP 1875 H=1875:00 +CPU= 39.022
  06:06:15 STEP 1876 H=1876:00 +CPU= 24.423
  06:06:39 STEP 1877 H=1877:00 +CPU= 23.849
  06:07:17 STEP 1878 H=1878:00 +CPU= 38.801
  06:07:41 STEP 1879 H=1879:00 +CPU= 23.280
  06:08:05 STEP 1880 H=1880:00 +CPU= 24.206
  06:08:42 STEP 1881 H=1881:00 +CPU= 37.124
  06:09:07 STEP 1882 H=1882:00 +CPU= 24.335
  06:09:30 STEP 1883 H=1883:00 +CPU= 23.424
  06:10:12 STEP 1884 H=1884:00 +CPU= 41.798
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001884
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001884
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001884
  06:10:36 STEP 1885 H=1885:00 +CPU= 23.674
  06:11:01 STEP 1886 H=1886:00 +CPU= 25.252
  06:11:40 STEP 1887 H=1887:00 +CPU= 39.107
  06:12:06 STEP 1888 H=1888:00 +CPU= 25.366
  06:12:30 STEP 1889 H=1889:00 +CPU= 24.602
  06:13:09 STEP 1890 H=1890:00 +CPU= 38.737
  06:13:33 STEP 1891 H=1891:00 +CPU= 24.189
  06:13:59 STEP 1892 H=1892:00 +CPU= 25.075
  06:14:37 STEP 1893 H=1893:00 +CPU= 38.274
  06:15:02 STEP 1894 H=1894:00 +CPU= 24.784
  06:15:26 STEP 1895 H=1895:00 +CPU= 24.585
  06:16:11 STEP 1896 H=1896:00 +CPU= 44.667
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001896
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001896
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001872
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001884
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001884
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001884
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_78.zip
Uploading the intermediate file: upload_file_78.zip
  06:16:35 STEP 1897 H=1897:00 +CPU= 23.656
  06:17:00 STEP 1898 H=1898:00 +CPU= 24.677
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0769
(argv4) batchid: 984
(argv5) wuid: 12201413
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mydisks/a/boinc_lib/slots/30
Project directory is: /mydisks/a/boinc_lib/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413
..mkdir for temp folder for results failed
Copying: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_6ccc10347dc5194afae77b400e006e0d to: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_0769_2015050100_123_984_12201413.zip
Unzipping the namelist zip file: /mydisks/a/boinc_lib/slots/30/oifs_43r3_ps_0769_2015050100_123_984_12201413.zip
ic_ancil_file: ic_ancil_12201413
ifsdata_file: ifsdata_12201413
climate_data_file: clim_data_12201413
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c3a7a1d21fa2cbee2d96b97baf803571 to: /mydisks/a/boinc_lib/slots/30/ic_ancil_12201413.zip
Unzipping the IC ancils zip file: /mydisks/a/boinc_lib/slots/30/ic_ancil_12201413.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c0a55765f2ed03ab43d4fbe2239c9369 to: /mydisks/a/boinc_lib/slots/30/ifsdata/ifsdata_12201413.zip
Unzipping the ifsdata_zip file: /mydisks/a/boinc_lib/slots/30/ifsdata/ifsdata_12201413.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mydisks/a/boinc_lib/slots/30/159l_2/clim_data_12201413.zip
Unzipping the climate data zip file: /mydisks/a/boinc_lib/slots/30/159l_2/clim_data_12201413.zip
Checking for progress XML file: /mydisks/a/boinc_lib/slots/30/progress_file_12201413.xml
Opened progress file ok : /mydisks/a/boinc_lib/slots/30/progress_file_12201413.xml
-- Model is restarting --
Adjusting last_iter, 1899, to previous model restart step.
Creating progress file: /mydisks/a/boinc_lib/slots/30/progress_file_12201413.xml
last_cpu_time: 58076
upload_file_number: 79
last_iter: 1895
last_upload: 6825600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781
The child process has been launched with process id: 656
Executing the command: /mydisks/a/boinc_lib/slots/30/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:amigos:1:1:656:656] [20230125:062011:1674656411:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  06:20:52 STEP 1896 H=1896:00 +CPU= 37.969
  06:21:12 STEP 1897 H=1897:00 +CPU= 20.428
  06:21:34 STEP 1898 H=1898:00 +CPU= 21.367
  06:22:07 STEP 1899 H=1899:00 +CPU= 33.562
  06:22:29 STEP 1900 H=1900:00 +CPU= 21.857
  06:22:50 STEP 1901 H=1901:00 +CPU= 21.139
  06:23:25 STEP 1902 H=1902:00 +CPU= 34.288
  06:23:45 STEP 1903 H=1903:00 +CPU= 20.571
  06:24:07 STEP 1904 H=1904:00 +CPU= 22.022
  06:24:41 STEP 1905 H=1905:00 +CPU= 33.346
  06:25:02 STEP 1906 H=1906:00 +CPU= 21.399
  06:25:24 STEP 1907 H=1907:00 +CPU= 21.961
  06:26:04 STEP 1908 H=1908:00 +CPU= 39.693
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001908
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001908
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001908
  06:26:26 STEP 1909 H=1909:00 +CPU= 22.279
  06:26:50 STEP 1910 H=1910:00 +CPU= 23.425
  06:27:23 STEP 1911 H=1911:00 +CPU= 33.802
  06:27:45 STEP 1912 H=1912:00 +CPU= 21.631
  06:28:07 STEP 1913 H=1913:00 +CPU= 21.481
  06:28:44 STEP 1914 H=1914:00 +CPU= 36.989
  06:29:07 STEP 1915 H=1915:00 +CPU= 23.314
  06:29:29 STEP 1916 H=1916:00 +CPU= 22.287
  06:30:03 STEP 1917 H=1917:00 +CPU= 34.088
  06:30:26 STEP 1918 H=1918:00 +CPU= 22.246
  06:30:47 STEP 1919 H=1919:00 +CPU= 21.618
  06:31:27 STEP 1920 H=1920:00 +CPU= 39.367
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMGGhq0f+001920
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMSHhq0f+001920
Moving to projects directory: /mydisks/a/boinc_lib/slots/30/ICMUAhq0f+001920
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001896
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMGGhq0f+001908
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMSHhq0f+001908
Adding to the zip: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_12201413/ICMUAhq0f+001908
Zipping up the intermediate file: /mydisks/a/boinc_lib/projects/climateprediction.net/oifs_43r3_ps_0769_2015050100_123_984_12201413_1_r220195781_79.zip
Uploading the intermediate file: upload_file_79.zip
  06:31:48 STEP 1921 H=1921:00 +CPU= 20.801
  06:32:09 STEP 1922 H=1922:00 +CPU= 21.556
  06:32:44 STEP 1923 H=1923:00 +CPU= 34.507
  06:33:06 STEP 1924 H=1924:00 +CPU= 21.952
  06:33:27 STEP 1925 H=1925:00 +CPU= 20.909
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:40:50 1517479 22302847 oifs_43r3_ps_0769_2015050100_123_984_12201413_1 6,652,800 56,612 0.0085
09 May 2023 16:40:14 1517479 22302847 oifs_43r3_ps_0769_2015050100_123_984_12201413_1 2,592,000 23,031 0.0089
09 May 2023 16:39:31 1517479 22302847 oifs_43r3_ps_0769_2015050100_123_984_12201413_1 1,728,000 15,297 0.0089
09 May 2023 16:39:23 1517479 22302847 oifs_43r3_ps_0769_2015050100_123_984_12201413_1 864,000 7,659 0.0089
09 May 2023 16:39:23 1517479 22302847 oifs_43r3_ps_0769_2015050100_123_984_12201413_1 864,000 7,659 0.0089


©2024 cpdn.org