climateprediction.net home page
Task 22268300

Task 22268300

Name oifs_43r3_ps_0506_1997050100_123_966_12183150_0
Workunit 12183150
Created 22 Dec 2022, 13:46:23 UTC
Sent 25 Dec 2022, 22:31:10 UTC
Report deadline 24 Jan 2023, 22:31:10 UTC
Received 26 Dec 2022, 15:35:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1538662
Run time 12 hours 24 min 35 sec
CPU time 11 hours 22 min 54 sec
Validate state Invalid
Credit 8,588.16
Device peak FLOPS 7.18 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.74 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,180.10 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
t/oifs_43r3_ps_12183150/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_59.zip
Uploading the intermediate file: upload_file_59.zip
  12:17:59 STEP 1441 H=1441:00 +CPU= 17.463
  12:18:18 STEP 1442 H=1442:00 +CPU= 17.573
  12:18:47 STEP 1443 H=1443:00 +CPU= 26.868
  12:19:05 STEP 1444 H=1444:00 +CPU= 17.521
  12:19:24 STEP 1445 H=1445:00 +CPU= 16.885
  12:19:54 STEP 1446 H=1446:00 +CPU= 27.233
  12:20:13 STEP 1447 H=1447:00 +CPU= 16.940
  12:20:33 STEP 1448 H=1448:00 +CPU= 17.874
  12:21:05 STEP 1449 H=1449:00 +CPU= 28.228
  12:21:26 STEP 1450 H=1450:00 +CPU= 18.739
  12:21:46 STEP 1451 H=1451:00 +CPU= 18.132
  12:22:24 STEP 1452 H=1452:00 +CPU= 32.462
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001452
  12:22:43 STEP 1453 H=1453:00 +CPU= 18.110
  12:23:04 STEP 1454 H=1454:00 +CPU= 18.911
  12:23:34 STEP 1455 H=1455:00 +CPU= 27.654
  12:23:53 STEP 1456 H=1456:00 +CPU= 17.473
  12:24:12 STEP 1457 H=1457:00 +CPU= 17.264
  12:24:44 STEP 1458 H=1458:00 +CPU= 27.810
  12:25:02 STEP 1459 H=1459:00 +CPU= 17.278
  12:25:22 STEP 1460 H=1460:00 +CPU= 17.483
  12:25:52 STEP 1461 H=1461:00 +CPU= 26.877
  12:26:12 STEP 1462 H=1462:00 +CPU= 17.406
  12:26:32 STEP 1463 H=1463:00 +CPU= 18.265
  12:27:11 STEP 1464 H=1464:00 +CPU= 33.471
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_60.zip
Uploading the intermediate file: upload_file_60.zip
  12:27:32 STEP 1465 H=1465:00 +CPU= 18.172
Uploading trickle at timestep: 5270400
  12:27:53 STEP 1466 H=1466:00 +CPU= 19.125
  12:28:25 STEP 1467 H=1467:00 +CPU= 28.225
  12:28:46 STEP 1468 H=1468:00 +CPU= 18.310
  12:29:06 STEP 1469 H=1469:00 +CPU= 18.221
  12:29:36 STEP 1470 H=1470:00 +CPU= 28.142
  12:29:55 STEP 1471 H=1471:00 +CPU= 17.386
  12:30:14 STEP 1472 H=1472:00 +CPU= 17.465
  12:30:44 STEP 1473 H=1473:00 +CPU= 27.391
  12:31:02 STEP 1474 H=1474:00 +CPU= 17.152
  12:31:22 STEP 1475 H=1475:00 +CPU= 16.803
  12:31:57 STEP 1476 H=1476:00 +CPU= 30.291
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001476
  12:32:17 STEP 1477 H=1477:00 +CPU= 17.821
  12:32:39 STEP 1478 H=1478:00 +CPU= 18.449
  12:33:10 STEP 1479 H=1479:00 +CPU= 28.941
  12:33:31 STEP 1480 H=1480:00 +CPU= 18.346
  12:33:51 STEP 1481 H=1481:00 +CPU= 18.024
  12:34:24 STEP 1482 H=1482:00 +CPU= 29.699
  12:34:44 STEP 1483 H=1483:00 +CPU= 18.262
  12:35:03 STEP 1484 H=1484:00 +CPU= 17.672
  12:35:34 STEP 1485 H=1485:00 +CPU= 26.943
  12:35:53 STEP 1486 H=1486:00 +CPU= 17.823
  12:36:12 STEP 1487 H=1487:00 +CPU= 17.629
  12:36:46 STEP 1488 H=1488:00 +CPU= 31.934
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_61.zip
Uploading the intermediate file: upload_file_61.zip
  12:37:05 STEP 1489 H=1489:00 +CPU= 16.806
  12:37:24 STEP 1490 H=1490:00 +CPU= 17.436
  12:37:56 STEP 1491 H=1491:00 +CPU= 27.740
  12:38:16 STEP 1492 H=1492:00 +CPU= 18.044
  12:38:36 STEP 1493 H=1493:00 +CPU= 18.038
  12:39:09 STEP 1494 H=1494:00 +CPU= 30.127
  12:39:30 STEP 1495 H=1495:00 +CPU= 18.334
  12:39:52 STEP 1496 H=1496:00 +CPU= 20.942
  12:40:22 STEP 1497 H=1497:00 +CPU= 26.779
  12:40:42 STEP 1498 H=1498:00 +CPU= 18.035
  12:41:02 STEP 1499 H=1499:00 +CPU= 17.231
  12:41:34 STEP 1500 H=1500:00 +CPU= 30.051
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001500
  12:41:54 STEP 1501 H=1501:00 +CPU= 17.933
  12:42:13 STEP 1502 H=1502:00 +CPU= 17.007
  12:42:42 STEP 1503 H=1503:00 +CPU= 26.998
  12:43:01 STEP 1504 H=1504:00 +CPU= 17.729
  12:43:20 STEP 1505 H=1505:00 +CPU= 16.907
  12:43:51 STEP 1506 H=1506:00 +CPU= 28.297
  12:44:11 STEP 1507 H=1507:00 +CPU= 17.364
  12:44:32 STEP 1508 H=1508:00 +CPU= 18.599
  12:45:04 STEP 1509 H=1509:00 +CPU= 28.623
  12:45:25 STEP 1510 H=1510:00 +CPU= 19.009
  12:45:46 STEP 1511 H=1511:00 +CPU= 18.705
  12:46:22 STEP 1512 H=1512:00 +CPU= 32.612
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_62.zip
Uploading the intermediate file: upload_file_62.zip
  12:46:42 STEP 1513 H=1513:00 +CPU= 17.091
  12:47:01 STEP 1514 H=1514:00 +CPU= 17.687
  12:47:31 STEP 1515 H=1515:00 +CPU= 27.817
  12:47:50 STEP 1516 H=1516:00 +CPU= 17.277
  12:48:08 STEP 1517 H=1517:00 +CPU= 16.722
  12:48:38 STEP 1518 H=1518:00 +CPU= 27.255
  12:48:57 STEP 1519 H=1519:00 +CPU= 17.010
  12:49:17 STEP 1520 H=1520:00 +CPU= 17.623
  12:49:48 STEP 1521 H=1521:00 +CPU= 27.538
  12:50:08 STEP 1522 H=1522:00 +CPU= 18.667
  12:50:28 STEP 1523 H=1523:00 +CPU= 17.973
  12:51:04 STEP 1524 H=1524:00 +CPU= 31.950
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001524
  12:51:24 STEP 1525 H=1525:00 +CPU= 17.968
  12:51:45 STEP 1526 H=1526:00 +CPU= 19.002
  12:52:16 STEP 1527 H=1527:00 +CPU= 27.777
  12:52:36 STEP 1528 H=1528:00 +CPU= 18.711
  12:52:54 STEP 1529 H=1529:00 +CPU= 16.783
  12:53:25 STEP 1530 H=1530:00 +CPU= 27.477
  12:53:45 STEP 1531 H=1531:00 +CPU= 17.614
  12:54:04 STEP 1532 H=1532:00 +CPU= 17.514
  12:54:34 STEP 1533 H=1533:00 +CPU= 27.153
  12:54:54 STEP 1534 H=1534:00 +CPU= 17.251
  12:55:15 STEP 1535 H=1535:00 +CPU= 18.061
  12:55:53 STEP 1536 H=1536:00 +CPU= 33.332
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_63.zip
Uploading the intermediate file: upload_file_63.zip
  12:56:12 STEP 1537 H=1537:00 +CPU= 17.993
  12:56:33 STEP 1538 H=1538:00 +CPU= 19.043
  12:57:05 STEP 1539 H=1539:00 +CPU= 29.052
  12:57:26 STEP 1540 H=1540:00 +CPU= 18.610
  12:57:45 STEP 1541 H=1541:00 +CPU= 17.582
  12:58:17 STEP 1542 H=1542:00 +CPU= 28.199
  12:58:37 STEP 1543 H=1543:00 +CPU= 19.517
  12:58:56 STEP 1544 H=1544:00 +CPU= 18.026
  12:59:26 STEP 1545 H=1545:00 +CPU= 27.127
  12:59:45 STEP 1546 H=1546:00 +CPU= 17.373
  13:00:05 STEP 1547 H=1547:00 +CPU= 17.217
  13:00:39 STEP 1548 H=1548:00 +CPU= 30.389
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001548
  13:00:59 STEP 1549 H=1549:00 +CPU= 17.740
  13:01:20 STEP 1550 H=1550:00 +CPU= 18.135
  13:01:51 STEP 1551 H=1551:00 +CPU= 28.397
  13:02:12 STEP 1552 H=1552:00 +CPU= 18.692
  13:02:32 STEP 1553 H=1553:00 +CPU= 18.149
  13:03:05 STEP 1554 H=1554:00 +CPU= 29.207
  13:03:24 STEP 1555 H=1555:00 +CPU= 18.216
  13:03:45 STEP 1556 H=1556:00 +CPU= 18.111
  13:04:15 STEP 1557 H=1557:00 +CPU= 27.961
  13:04:35 STEP 1558 H=1558:00 +CPU= 18.375
  13:04:54 STEP 1559 H=1559:00 +CPU= 17.233
  13:05:29 STEP 1560 H=1560:00 +CPU= 31.601
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_64.zip
Uploading the intermediate file: upload_file_64.zip
  13:05:48 STEP 1561 H=1561:00 +CPU= 16.752
  13:06:07 STEP 1562 H=1562:00 +CPU= 17.598
  13:06:37 STEP 1563 H=1563:00 +CPU= 27.142
  13:06:58 STEP 1564 H=1564:00 +CPU= 17.911
  13:07:19 STEP 1565 H=1565:00 +CPU= 17.821
  13:07:51 STEP 1566 H=1566:00 +CPU= 28.969
  13:08:12 STEP 1567 H=1567:00 +CPU= 18.787
  13:08:32 STEP 1568 H=1568:00 +CPU= 18.334
  13:09:04 STEP 1569 H=1569:00 +CPU= 29.096
  13:09:23 STEP 1570 H=1570:00 +CPU= 18.298
  13:09:43 STEP 1571 H=1571:00 +CPU= 17.294
  13:10:16 STEP 1572 H=1572:00 +CPU= 30.307
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001572
  13:10:34 STEP 1573 H=1573:00 +CPU= 17.286
  13:10:54 STEP 1574 H=1574:00 +CPU= 18.014
  13:11:23 STEP 1575 H=1575:00 +CPU= 26.656
  13:11:42 STEP 1576 H=1576:00 +CPU= 17.248
  13:12:01 STEP 1577 H=1577:00 +CPU= 16.885
  13:12:32 STEP 1578 H=1578:00 +CPU= 28.458
  13:12:51 STEP 1579 H=1579:00 +CPU= 17.167
  13:13:10 STEP 1580 H=1580:00 +CPU= 18.246
  13:13:42 STEP 1581 H=1581:00 +CPU= 28.677
  13:14:03 STEP 1582 H=1582:00 +CPU= 18.700
  13:14:25 STEP 1583 H=1583:00 +CPU= 19.605
  13:15:01 STEP 1584 H=1584:00 +CPU= 32.874
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_65.zip
Uploading the intermediate file: upload_file_65.zip
  13:15:20 STEP 1585 H=1585:00 +CPU= 17.163
  13:15:40 STEP 1586 H=1586:00 +CPU= 18.381
  13:16:12 STEP 1587 H=1587:00 +CPU= 26.475
  13:16:32 STEP 1588 H=1588:00 +CPU= 17.511
  13:16:50 STEP 1589 H=1589:00 +CPU= 16.750
  13:17:22 STEP 1590 H=1590:00 +CPU= 27.600
  13:17:42 STEP 1591 H=1591:00 +CPU= 16.741
  13:18:02 STEP 1592 H=1592:00 +CPU= 17.699
  13:18:32 STEP 1593 H=1593:00 +CPU= 27.466
  13:18:53 STEP 1594 H=1594:00 +CPU= 18.563
  13:19:12 STEP 1595 H=1595:00 +CPU= 17.724
  13:19:48 STEP 1596 H=1596:00 +CPU= 33.259
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001596
  13:20:08 STEP 1597 H=1597:00 +CPU= 17.870
  13:20:29 STEP 1598 H=1598:00 +CPU= 18.816
  13:21:01 STEP 1599 H=1599:00 +CPU= 28.421
  13:21:20 STEP 1600 H=1600:00 +CPU= 18.125
  13:21:39 STEP 1601 H=1601:00 +CPU= 16.737
  13:22:10 STEP 1602 H=1602:00 +CPU= 28.754
  13:22:29 STEP 1603 H=1603:00 +CPU= 16.966
  13:22:47 STEP 1604 H=1604:00 +CPU= 16.704
  13:23:15 STEP 1605 H=1605:00 +CPU= 26.024
  13:23:33 STEP 1606 H=1606:00 +CPU= 16.525
  13:23:50 STEP 1607 H=1607:00 +CPU= 16.264
  13:24:25 STEP 1608 H=1608:00 +CPU= 31.089
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_66.zip
Uploading the intermediate file: upload_file_66.zip
  13:24:43 STEP 1609 H=1609:00 +CPU= 16.987
  13:25:02 STEP 1610 H=1610:00 +CPU= 17.367
  13:25:31 STEP 1611 H=1611:00 +CPU= 27.003
  13:25:49 STEP 1612 H=1612:00 +CPU= 17.289
  13:26:07 STEP 1613 H=1613:00 +CPU= 16.756
  13:26:37 STEP 1614 H=1614:00 +CPU= 27.588
  13:26:55 STEP 1615 H=1615:00 +CPU= 16.581
  13:27:14 STEP 1616 H=1616:00 +CPU= 17.383
  13:27:44 STEP 1617 H=1617:00 +CPU= 26.925
  13:28:02 STEP 1618 H=1618:00 +CPU= 17.315
  13:28:21 STEP 1619 H=1619:00 +CPU= 17.008
  13:28:53 STEP 1620 H=1620:00 +CPU= 30.058
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001620
  13:29:12 STEP 1621 H=1621:00 +CPU= 16.856
  13:29:30 STEP 1622 H=1622:00 +CPU= 17.086
  13:29:59 STEP 1623 H=1623:00 +CPU= 26.677
  13:30:18 STEP 1624 H=1624:00 +CPU= 17.010
  13:30:37 STEP 1625 H=1625:00 +CPU= 16.782
  13:31:06 STEP 1626 H=1626:00 +CPU= 27.417
  13:31:23 STEP 1627 H=1627:00 +CPU= 16.413
  13:31:42 STEP 1628 H=1628:00 +CPU= 17.360
  13:32:11 STEP 1629 H=1629:00 +CPU= 26.864
  13:32:30 STEP 1630 H=1630:00 +CPU= 17.252
  13:32:48 STEP 1631 H=1631:00 +CPU= 17.294
  13:33:23 STEP 1632 H=1632:00 +CPU= 31.868
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_67.zip
Uploading the intermediate file: upload_file_67.zip
  13:33:40 STEP 1633 H=1633:00 +CPU= 16.580
  13:33:59 STEP 1634 H=1634:00 +CPU= 17.191
  13:34:28 STEP 1635 H=1635:00 +CPU= 26.727
  13:34:46 STEP 1636 H=1636:00 +CPU= 16.852
  13:35:05 STEP 1637 H=1637:00 +CPU= 16.953
  13:35:34 STEP 1638 H=1638:00 +CPU= 27.068
  13:35:52 STEP 1639 H=1639:00 +CPU= 16.498
  13:36:11 STEP 1640 H=1640:00 +CPU= 17.186
  13:36:40 STEP 1641 H=1641:00 +CPU= 27.036
  13:36:58 STEP 1642 H=1642:00 +CPU= 16.981
  13:37:16 STEP 1643 H=1643:00 +CPU= 16.418
  13:37:48 STEP 1644 H=1644:00 +CPU= 29.395
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001644
  13:38:06 STEP 1645 H=1645:00 +CPU= 16.175
  13:38:24 STEP 1646 H=1646:00 +CPU= 16.851
  13:38:53 STEP 1647 H=1647:00 +CPU= 26.808
  13:39:12 STEP 1648 H=1648:00 +CPU= 17.502
  13:39:30 STEP 1649 H=1649:00 +CPU= 17.232
  13:40:00 STEP 1650 H=1650:00 +CPU= 27.598
  13:40:18 STEP 1651 H=1651:00 +CPU= 16.909
  13:40:38 STEP 1652 H=1652:00 +CPU= 17.692
  13:41:07 STEP 1653 H=1653:00 +CPU= 26.951
  13:41:26 STEP 1654 H=1654:00 +CPU= 17.440
  13:41:45 STEP 1655 H=1655:00 +CPU= 17.360
  13:42:19 STEP 1656 H=1656:00 +CPU= 32.122
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_68.zip
Uploading the intermediate file: upload_file_68.zip
  13:42:38 STEP 1657 H=1657:00 +CPU= 16.653
  13:42:56 STEP 1658 H=1658:00 +CPU= 17.249
  13:43:25 STEP 1659 H=1659:00 +CPU= 26.844
  13:43:43 STEP 1660 H=1660:00 +CPU= 17.316
  13:44:02 STEP 1661 H=1661:00 +CPU= 17.015
  13:44:32 STEP 1662 H=1662:00 +CPU= 27.405
  13:44:49 STEP 1663 H=1663:00 +CPU= 16.522
  13:45:08 STEP 1664 H=1664:00 +CPU= 17.349
  13:45:37 STEP 1665 H=1665:00 +CPU= 26.409
  13:45:55 STEP 1666 H=1666:00 +CPU= 17.297
  13:46:13 STEP 1667 H=1667:00 +CPU= 16.897
  13:46:46 STEP 1668 H=1668:00 +CPU= 29.977
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001668
  13:47:05 STEP 1669 H=1669:00 +CPU= 16.829
  13:47:23 STEP 1670 H=1670:00 +CPU= 16.936
  13:47:52 STEP 1671 H=1671:00 +CPU= 26.860
  13:48:11 STEP 1672 H=1672:00 +CPU= 17.466
  13:48:29 STEP 1673 H=1673:00 +CPU= 16.660
  13:48:58 STEP 1674 H=1674:00 +CPU= 27.199
  13:49:16 STEP 1675 H=1675:00 +CPU= 16.962
  13:49:34 STEP 1676 H=1676:00 +CPU= 17.118
  13:50:03 STEP 1677 H=1677:00 +CPU= 26.588
  13:50:22 STEP 1678 H=1678:00 +CPU= 17.365
  13:50:40 STEP 1679 H=1679:00 +CPU= 16.710
  13:51:14 STEP 1680 H=1680:00 +CPU= 31.083
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_69.zip
Uploading the intermediate file: upload_file_69.zip
  13:51:32 STEP 1681 H=1681:00 +CPU= 16.066
  13:51:50 STEP 1682 H=1682:00 +CPU= 16.691
  13:52:18 STEP 1683 H=1683:00 +CPU= 26.467
  13:52:37 STEP 1684 H=1684:00 +CPU= 17.320
  13:52:55 STEP 1685 H=1685:00 +CPU= 17.060
  13:53:25 STEP 1686 H=1686:00 +CPU= 27.875
  13:53:44 STEP 1687 H=1687:00 +CPU= 16.997
  13:54:02 STEP 1688 H=1688:00 +CPU= 17.380
  13:54:31 STEP 1689 H=1689:00 +CPU= 26.868
  13:54:50 STEP 1690 H=1690:00 +CPU= 17.420
  13:55:08 STEP 1691 H=1691:00 +CPU= 16.376
  13:55:40 STEP 1692 H=1692:00 +CPU= 30.013
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001692
  13:55:58 STEP 1693 H=1693:00 +CPU= 16.857
  13:56:17 STEP 1694 H=1694:00 +CPU= 17.086
  13:56:45 STEP 1695 H=1695:00 +CPU= 26.238
  13:57:04 STEP 1696 H=1696:00 +CPU= 17.376
  13:57:22 STEP 1697 H=1697:00 +CPU= 16.912
  13:57:51 STEP 1698 H=1698:00 +CPU= 26.744
  13:58:09 STEP 1699 H=1699:00 +CPU= 16.779
  13:58:28 STEP 1700 H=1700:00 +CPU= 17.267
  13:58:56 STEP 1701 H=1701:00 +CPU= 26.193
  13:59:15 STEP 1702 H=1702:00 +CPU= 17.171
  13:59:33 STEP 1703 H=1703:00 +CPU= 17.060
  14:00:08 STEP 1704 H=1704:00 +CPU= 31.121
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_70.zip
Uploading the intermediate file: upload_file_70.zip
  14:00:25 STEP 1705 H=1705:00 +CPU= 15.870
Uploading trickle at timestep: 6134400
  14:00:43 STEP 1706 H=1706:00 +CPU= 16.358
  14:01:12 STEP 1707 H=1707:00 +CPU= 25.803
  14:01:30 STEP 1708 H=1708:00 +CPU= 16.815
  14:01:48 STEP 1709 H=1709:00 +CPU= 16.678
  14:02:16 STEP 1710 H=1710:00 +CPU= 26.915
  14:02:34 STEP 1711 H=1711:00 +CPU= 16.662
  14:02:53 STEP 1712 H=1712:00 +CPU= 17.533
  14:03:23 STEP 1713 H=1713:00 +CPU= 27.152
  14:03:42 STEP 1714 H=1714:00 +CPU= 17.222
  14:03:59 STEP 1715 H=1715:00 +CPU= 16.652
  14:04:32 STEP 1716 H=1716:00 +CPU= 30.126
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001716
  14:04:50 STEP 1717 H=1717:00 +CPU= 16.735
  14:05:08 STEP 1718 H=1718:00 +CPU= 17.060
  14:05:37 STEP 1719 H=1719:00 +CPU= 26.460
  14:05:56 STEP 1720 H=1720:00 +CPU= 17.102
  14:06:13 STEP 1721 H=1721:00 +CPU= 16.492
  14:06:42 STEP 1722 H=1722:00 +CPU= 26.818
  14:07:00 STEP 1723 H=1723:00 +CPU= 16.451
  14:07:19 STEP 1724 H=1724:00 +CPU= 17.217
  14:07:48 STEP 1725 H=1725:00 +CPU= 26.477
  14:08:06 STEP 1726 H=1726:00 +CPU= 16.812
  14:08:25 STEP 1727 H=1727:00 +CPU= 17.162
  14:08:59 STEP 1728 H=1728:00 +CPU= 31.663
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_71.zip
Uploading the intermediate file: upload_file_71.zip
  14:09:16 STEP 1729 H=1729:00 +CPU= 16.032
  14:09:34 STEP 1730 H=1730:00 +CPU= 16.554
  14:10:02 STEP 1731 H=1731:00 +CPU= 25.405
  14:10:19 STEP 1732 H=1732:00 +CPU= 16.158
  14:10:37 STEP 1733 H=1733:00 +CPU= 16.188
  14:11:06 STEP 1734 H=1734:00 +CPU= 26.970
  14:11:24 STEP 1735 H=1735:00 +CPU= 16.504
  14:11:42 STEP 1736 H=1736:00 +CPU= 17.046
  14:12:11 STEP 1737 H=1737:00 +CPU= 26.438
  14:12:30 STEP 1738 H=1738:00 +CPU= 17.184
  14:12:48 STEP 1739 H=1739:00 +CPU= 16.771
  14:13:20 STEP 1740 H=1740:00 +CPU= 29.989
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001740
  14:13:38 STEP 1741 H=1741:00 +CPU= 16.525
  14:13:57 STEP 1742 H=1742:00 +CPU= 17.153
  14:14:25 STEP 1743 H=1743:00 +CPU= 26.156
  14:14:43 STEP 1744 H=1744:00 +CPU= 17.211
  14:15:01 STEP 1745 H=1745:00 +CPU= 16.324
  14:15:30 STEP 1746 H=1746:00 +CPU= 27.085
  14:15:48 STEP 1747 H=1747:00 +CPU= 16.466
  14:16:06 STEP 1748 H=1748:00 +CPU= 17.125
  14:16:34 STEP 1749 H=1749:00 +CPU= 26.074
  14:16:52 STEP 1750 H=1750:00 +CPU= 16.690
  14:17:10 STEP 1751 H=1751:00 +CPU= 16.856
  14:17:44 STEP 1752 H=1752:00 +CPU= 31.109
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_72.zip
Uploading the intermediate file: upload_file_72.zip
  14:18:02 STEP 1753 H=1753:00 +CPU= 16.609
  14:18:20 STEP 1754 H=1754:00 +CPU= 17.162
  14:18:49 STEP 1755 H=1755:00 +CPU= 26.009
  14:19:07 STEP 1756 H=1756:00 +CPU= 17.105
  14:19:26 STEP 1757 H=1757:00 +CPU= 16.472
  14:19:55 STEP 1758 H=1758:00 +CPU= 26.888
  14:20:12 STEP 1759 H=1759:00 +CPU= 16.383
  14:20:31 STEP 1760 H=1760:00 +CPU= 16.949
  14:20:59 STEP 1761 H=1761:00 +CPU= 25.997
  14:21:18 STEP 1762 H=1762:00 +CPU= 17.261
  14:21:36 STEP 1763 H=1763:00 +CPU= 16.350
  14:22:07 STEP 1764 H=1764:00 +CPU= 29.340
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001764
  14:22:24 STEP 1765 H=1765:00 +CPU= 16.213
  14:22:43 STEP 1766 H=1766:00 +CPU= 16.782
  14:23:11 STEP 1767 H=1767:00 +CPU= 26.004
  14:23:29 STEP 1768 H=1768:00 +CPU= 17.095
  14:23:47 STEP 1769 H=1769:00 +CPU= 16.393
  14:24:16 STEP 1770 H=1770:00 +CPU= 27.051
  14:24:34 STEP 1771 H=1771:00 +CPU= 16.665
  14:24:52 STEP 1772 H=1772:00 +CPU= 16.983
  14:25:20 STEP 1773 H=1773:00 +CPU= 25.804
  14:25:39 STEP 1774 H=1774:00 +CPU= 17.168
  14:25:57 STEP 1775 H=1775:00 +CPU= 16.689
  14:26:30 STEP 1776 H=1776:00 +CPU= 31.215
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_73.zip
Uploading the intermediate file: upload_file_73.zip
  14:26:48 STEP 1777 H=1777:00 +CPU= 16.296
  14:27:06 STEP 1778 H=1778:00 +CPU= 16.945
  14:27:34 STEP 1779 H=1779:00 +CPU= 25.890
  14:27:52 STEP 1780 H=1780:00 +CPU= 16.779
  14:28:10 STEP 1781 H=1781:00 +CPU= 16.033
  14:28:38 STEP 1782 H=1782:00 +CPU= 26.345
  14:28:56 STEP 1783 H=1783:00 +CPU= 16.400
  14:29:15 STEP 1784 H=1784:00 +CPU= 17.162
  14:29:44 STEP 1785 H=1785:00 +CPU= 27.037
  14:30:03 STEP 1786 H=1786:00 +CPU= 17.655
  14:30:21 STEP 1787 H=1787:00 +CPU= 17.070
  14:30:55 STEP 1788 H=1788:00 +CPU= 30.461
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001788
  14:31:13 STEP 1789 H=1789:00 +CPU= 16.820
  14:31:32 STEP 1790 H=1790:00 +CPU= 17.662
  14:32:01 STEP 1791 H=1791:00 +CPU= 27.102
  14:32:21 STEP 1792 H=1792:00 +CPU= 17.806
  14:32:40 STEP 1793 H=1793:00 +CPU= 17.021
  14:33:11 STEP 1794 H=1794:00 +CPU= 27.958
  14:33:30 STEP 1795 H=1795:00 +CPU= 17.361
  14:33:50 STEP 1796 H=1796:00 +CPU= 18.190
  14:34:21 STEP 1797 H=1797:00 +CPU= 27.805
  14:34:41 STEP 1798 H=1798:00 +CPU= 18.555
  14:35:02 STEP 1799 H=1799:00 +CPU= 18.769
  14:35:38 STEP 1800 H=1800:00 +CPU= 33.084
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_74.zip
Uploading the intermediate file: upload_file_74.zip
  14:35:57 STEP 1801 H=1801:00 +CPU= 17.746
  14:36:17 STEP 1802 H=1802:00 +CPU= 18.421
  14:36:47 STEP 1803 H=1803:00 +CPU= 27.049
  14:37:07 STEP 1804 H=1804:00 +CPU= 18.763
  14:37:25 STEP 1805 H=1805:00 +CPU= 16.578
  14:37:55 STEP 1806 H=1806:00 +CPU= 27.321
  14:38:13 STEP 1807 H=1807:00 +CPU= 17.045
  14:38:33 STEP 1808 H=1808:00 +CPU= 17.337
  14:39:04 STEP 1809 H=1809:00 +CPU= 27.734
  14:39:24 STEP 1810 H=1810:00 +CPU= 18.570
  14:39:43 STEP 1811 H=1811:00 +CPU= 17.728
  14:40:20 STEP 1812 H=1812:00 +CPU= 32.252
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001812
  14:40:40 STEP 1813 H=1813:00 +CPU= 18.490
  14:40:59 STEP 1814 H=1814:00 +CPU= 19.286
  14:41:31 STEP 1815 H=1815:00 +CPU= 28.557
  14:41:50 STEP 1816 H=1816:00 +CPU= 17.731
  14:42:09 STEP 1817 H=1817:00 +CPU= 17.479
  14:42:39 STEP 1818 H=1818:00 +CPU= 27.657
  14:42:58 STEP 1819 H=1819:00 +CPU= 17.812
  14:43:17 STEP 1820 H=1820:00 +CPU= 17.509
  14:43:47 STEP 1821 H=1821:00 +CPU= 26.884
  14:44:07 STEP 1822 H=1822:00 +CPU= 17.039
  14:44:28 STEP 1823 H=1823:00 +CPU= 17.610
  14:45:05 STEP 1824 H=1824:00 +CPU= 33.277
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_75.zip
Uploading the intermediate file: upload_file_75.zip
  14:45:24 STEP 1825 H=1825:00 +CPU= 17.887
  14:45:45 STEP 1826 H=1826:00 +CPU= 18.284
  14:46:17 STEP 1827 H=1827:00 +CPU= 29.096
  14:46:37 STEP 1828 H=1828:00 +CPU= 19.051
  14:46:57 STEP 1829 H=1829:00 +CPU= 17.809
  14:47:29 STEP 1830 H=1830:00 +CPU= 29.447
  14:47:48 STEP 1831 H=1831:00 +CPU= 17.746
  14:48:08 STEP 1832 H=1832:00 +CPU= 17.814
  14:48:38 STEP 1833 H=1833:00 +CPU= 27.467
  14:48:57 STEP 1834 H=1834:00 +CPU= 17.426
  14:49:15 STEP 1835 H=1835:00 +CPU= 17.319
  14:49:49 STEP 1836 H=1836:00 +CPU= 30.342
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001836
  14:50:09 STEP 1837 H=1837:00 +CPU= 17.451
  14:50:28 STEP 1838 H=1838:00 +CPU= 17.986
  14:51:00 STEP 1839 H=1839:00 +CPU= 28.620
  14:51:21 STEP 1840 H=1840:00 +CPU= 18.893
  14:51:41 STEP 1841 H=1841:00 +CPU= 18.135
  14:52:14 STEP 1842 H=1842:00 +CPU= 30.894
  14:52:34 STEP 1843 H=1843:00 +CPU= 17.659
  14:52:55 STEP 1844 H=1844:00 +CPU= 17.705
  14:53:27 STEP 1845 H=1845:00 +CPU= 27.831
  14:53:46 STEP 1846 H=1846:00 +CPU= 18.644
  14:54:06 STEP 1847 H=1847:00 +CPU= 18.560
  14:54:41 STEP 1848 H=1848:00 +CPU= 31.697
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_76.zip
Uploading the intermediate file: upload_file_76.zip
  14:55:00 STEP 1849 H=1849:00 +CPU= 17.089
  14:55:20 STEP 1850 H=1850:00 +CPU= 17.272
  14:55:49 STEP 1851 H=1851:00 +CPU= 26.113
  14:56:09 STEP 1852 H=1852:00 +CPU= 17.589
  14:56:28 STEP 1853 H=1853:00 +CPU= 16.839
  14:57:00 STEP 1854 H=1854:00 +CPU= 28.382
  14:57:19 STEP 1855 H=1855:00 +CPU= 17.782
  14:57:39 STEP 1856 H=1856:00 +CPU= 19.152
  14:58:11 STEP 1857 H=1857:00 +CPU= 27.918
  14:58:31 STEP 1858 H=1858:00 +CPU= 17.285
  14:58:50 STEP 1859 H=1859:00 +CPU= 16.890
  14:59:23 STEP 1860 H=1860:00 +CPU= 30.161
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001860
  14:59:41 STEP 1861 H=1861:00 +CPU= 17.381
  15:00:00 STEP 1862 H=1862:00 +CPU= 17.165
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.512] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.749] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.749] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.749] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.749] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.749] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.749] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.828] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150023:1672056023:14598.855] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                   VDFINCR 
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150025:1672056025:14599.879] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:tolik-B450M-S2H:1:1:447803:447803] [20221226:150025:1672056025:14599.879] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FCD831  vdfincr_                  208  vdfincr.F90
oifs_43r3_model.e  0000000000FAE435  vdfmain_                 1199  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1997050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0506
(argv4) batchid: 966
(argv5) wuid: 12183150
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0e103bc688d70c9e3378216b420af460 to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0506_1997050100_123_966_12183150.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0506_1997050100_123_966_12183150.zip
ic_ancil_file: ic_ancil_12183150
ifsdata_file: ifsdata_12183150
climate_data_file: clim_data_12183150
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1aadebfd9d7a82641b0da3477d43112f to: /var/lib/boinc-client/slots/13/ic_ancil_12183150.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12183150.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d1f9771d7e73cfd5589c96a7b79b227a to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12183150.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12183150.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12183150.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12183150.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12183150.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12183150.xml
-- Model is restarting --
Adjusting last_iter, 1863, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12183150.xml
last_cpu_time: 39691
upload_file_number: 77
last_iter: 1847
last_upload: 6652800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088
The child process has been launched with process id: 5700
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:tolik-B450M-S2H:1:1:5700:5700] [20221226:180238:1672066958:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:04:03 STEP 1848 H=1848:00 +CPU= 33.517
  18:04:28 STEP 1849 H=1849:00 +CPU= 17.566
  18:04:48 STEP 1850 H=1850:00 +CPU= 18.460
  18:05:19 STEP 1851 H=1851:00 +CPU= 28.451
  18:05:38 STEP 1852 H=1852:00 +CPU= 17.794
  18:05:57 STEP 1853 H=1853:00 +CPU= 18.013
  18:06:28 STEP 1854 H=1854:00 +CPU= 29.932
  18:06:49 STEP 1855 H=1855:00 +CPU= 19.003
  18:07:10 STEP 1856 H=1856:00 +CPU= 19.785
  18:07:42 STEP 1857 H=1857:00 +CPU= 29.958
  18:08:03 STEP 1858 H=1858:00 +CPU= 20.182
  18:08:23 STEP 1859 H=1859:00 +CPU= 18.820
  18:09:03 STEP 1860 H=1860:00 +CPU= 33.731
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001860
  18:09:22 STEP 1861 H=1861:00 +CPU= 18.041
  18:09:41 STEP 1862 H=1862:00 +CPU= 18.355
  18:10:14 STEP 1863 H=1863:00 +CPU= 29.497
  18:10:35 STEP 1864 H=1864:00 +CPU= 19.130
  18:10:56 STEP 1865 H=1865:00 +CPU= 18.422
  18:11:30 STEP 1866 H=1866:00 +CPU= 30.241
  18:11:51 STEP 1867 H=1867:00 +CPU= 18.611
  18:12:12 STEP 1868 H=1868:00 +CPU= 18.934
  18:12:45 STEP 1869 H=1869:00 +CPU= 30.196
  18:13:07 STEP 1870 H=1870:00 +CPU= 19.738
  18:13:29 STEP 1871 H=1871:00 +CPU= 20.000
  18:14:22 STEP 1872 H=1872:00 +CPU= 34.867
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_77.zip
Uploading the intermediate file: upload_file_77.zip
  18:14:44 STEP 1873 H=1873:00 +CPU= 19.132
  18:15:06 STEP 1874 H=1874:00 +CPU= 19.885
  18:15:38 STEP 1875 H=1875:00 +CPU= 29.635
  18:16:00 STEP 1876 H=1876:00 +CPU= 19.790
  18:16:21 STEP 1877 H=1877:00 +CPU= 18.560
  18:16:56 STEP 1878 H=1878:00 +CPU= 30.237
  18:17:17 STEP 1879 H=1879:00 +CPU= 18.765
  18:17:39 STEP 1880 H=1880:00 +CPU= 19.246
  18:18:13 STEP 1881 H=1881:00 +CPU= 29.451
  18:18:34 STEP 1882 H=1882:00 +CPU= 19.071
  18:18:56 STEP 1883 H=1883:00 +CPU= 19.483
  18:19:34 STEP 1884 H=1884:00 +CPU= 34.479
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001884
  18:19:54 STEP 1885 H=1885:00 +CPU= 18.803
  18:20:16 STEP 1886 H=1886:00 +CPU= 20.080
  18:20:50 STEP 1887 H=1887:00 +CPU= 30.845
  18:21:12 STEP 1888 H=1888:00 +CPU= 19.951
  18:21:33 STEP 1889 H=1889:00 +CPU= 19.409
  18:22:06 STEP 1890 H=1890:00 +CPU= 29.959
  18:22:27 STEP 1891 H=1891:00 +CPU= 18.424
  18:22:48 STEP 1892 H=1892:00 +CPU= 19.118
  18:23:21 STEP 1893 H=1893:00 +CPU= 29.598
  18:23:43 STEP 1894 H=1894:00 +CPU= 19.235
  18:24:04 STEP 1895 H=1895:00 +CPU= 19.347
  18:24:45 STEP 1896 H=1896:00 +CPU= 35.566
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12183150/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0506_1997050100_123_966_12183150_0_r354080088_78.zip
Uploading the intermediate file: upload_file_78.zip
  18:25:07 STEP 1897 H=1897:00 +CPU= 19.191
  18:25:30 STEP 1898 H=1898:00 +CPU= 20.058
  18:26:05 STEP 1899 H=1899:00 +CPU= 30.934
  18:26:28 STEP 1900 H=1900:00 +CPU= 20.551
  18:26:50 STEP 1901 H=1901:00 +CPU= 19.995
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:56:50 1518633 22268300 oifs_43r3_ps_0506_1997050100_123_966_12183150_0 6,134,400 36,386 0.0059
09 May 2023 15:56:46 1518633 22268300 oifs_43r3_ps_0506_1997050100_123_966_12183150_0 5,270,400 31,313 0.0059
09 May 2023 15:56:11 1518633 22268300 oifs_43r3_ps_0506_1997050100_123_966_12183150_0 2,678,400 15,645 0.0058
09 May 2023 15:56:00 1518633 22268300 oifs_43r3_ps_0506_1997050100_123_966_12183150_0 1,382,400 8,047 0.0058


©2024 climateprediction.net