climateprediction.net home page
Task 22298306

Task 22298306

Name oifs_43r3_ps_0886_2018050100_123_987_12204530_2
Workunit 12204530
Created 15 Jan 2023, 20:46:45 UTC
Sent 17 Jan 2023, 23:23:22 UTC
Report deadline 18 Mar 2023, 23:23:22 UTC
Received 18 Jan 2023, 22:04:45 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1538236
Run time 21 hours 49 min 37 sec
CPU time 21 hours 45 min 22 sec
Validate state Invalid
Credit 14,515.20
Device peak FLOPS 5.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.54 MB
Peak swap size 4,978.05 MB
Peak disk usage 2,011.03 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
file_104.zip
  17:40:40 STEP 2521 H=2521:00 +CPU= 24.917
  17:41:05 STEP 2522 H=2522:00 +CPU= 24.675
  17:41:42 STEP 2523 H=2523:00 +CPU= 36.970
  17:42:06 STEP 2524 H=2524:00 +CPU= 24.143
  17:42:31 STEP 2525 H=2525:00 +CPU= 24.650
  17:43:12 STEP 2526 H=2526:00 +CPU= 41.282
  17:43:38 STEP 2527 H=2527:00 +CPU= 26.157
  17:44:02 STEP 2528 H=2528:00 +CPU= 24.089
  17:44:40 STEP 2529 H=2529:00 +CPU= 37.144
  17:45:03 STEP 2530 H=2530:00 +CPU= 23.478
  17:45:26 STEP 2531 H=2531:00 +CPU= 23.095
  17:46:07 STEP 2532 H=2532:00 +CPU= 40.997
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002532
  17:46:26 STEP 2533 H=2533:00 +CPU= 19.231
  17:46:51 STEP 2534 H=2534:00 +CPU= 24.451
  17:47:29 STEP 2535 H=2535:00 +CPU= 38.120
  17:47:53 STEP 2536 H=2536:00 +CPU= 24.017
  17:48:16 STEP 2537 H=2537:00 +CPU= 23.020
  17:48:55 STEP 2538 H=2538:00 +CPU= 38.745
  17:49:18 STEP 2539 H=2539:00 +CPU= 23.346
  17:49:42 STEP 2540 H=2540:00 +CPU= 23.670
  17:50:19 STEP 2541 H=2541:00 +CPU= 36.960
  17:50:45 STEP 2542 H=2542:00 +CPU= 25.968
  17:51:08 STEP 2543 H=2543:00 +CPU= 23.125
  17:51:52 STEP 2544 H=2544:00 +CPU= 44.353
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_105.zip
Uploading the intermediate file: upload_file_105.zip
  17:52:16 STEP 2545 H=2545:00 +CPU= 23.752
  17:52:39 STEP 2546 H=2546:00 +CPU= 23.044
  17:53:15 STEP 2547 H=2547:00 +CPU= 36.340
  17:53:40 STEP 2548 H=2548:00 +CPU= 24.491
  17:54:04 STEP 2549 H=2549:00 +CPU= 23.949
  17:54:43 STEP 2550 H=2550:00 +CPU= 39.183
  17:55:10 STEP 2551 H=2551:00 +CPU= 26.737
  17:55:36 STEP 2552 H=2552:00 +CPU= 26.146
  17:56:16 STEP 2553 H=2553:00 +CPU= 40.395
  17:56:41 STEP 2554 H=2554:00 +CPU= 24.763
  17:57:07 STEP 2555 H=2555:00 +CPU= 25.515
  17:57:50 STEP 2556 H=2556:00 +CPU= 43.258
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002556
  17:58:15 STEP 2557 H=2557:00 +CPU= 24.652
  17:58:41 STEP 2558 H=2558:00 +CPU= 26.513
  17:59:20 STEP 2559 H=2559:00 +CPU= 38.792
  17:59:42 STEP 2560 H=2560:00 +CPU= 21.719
  18:00:05 STEP 2561 H=2561:00 +CPU= 23.313
  18:00:42 STEP 2562 H=2562:00 +CPU= 37.072
  18:01:06 STEP 2563 H=2563:00 +CPU= 23.658
  18:01:27 STEP 2564 H=2564:00 +CPU= 21.155
  18:01:55 STEP 2565 H=2565:00 +CPU= 27.919
  18:02:18 STEP 2566 H=2566:00 +CPU= 23.155
  18:02:39 STEP 2567 H=2567:00 +CPU= 20.885
  18:03:24 STEP 2568 H=2568:00 +CPU= 44.641
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_106.zip
Uploading the intermediate file: upload_file_106.zip
  18:03:47 STEP 2569 H=2569:00 +CPU= 23.243
  18:04:12 STEP 2570 H=2570:00 +CPU= 25.101
  18:04:50 STEP 2571 H=2571:00 +CPU= 38.061
  18:05:16 STEP 2572 H=2572:00 +CPU= 25.523
  18:05:40 STEP 2573 H=2573:00 +CPU= 23.941
  18:06:17 STEP 2574 H=2574:00 +CPU= 37.654
  18:06:42 STEP 2575 H=2575:00 +CPU= 24.672
  18:07:07 STEP 2576 H=2576:00 +CPU= 24.655
  18:07:45 STEP 2577 H=2577:00 +CPU= 38.740
  18:08:11 STEP 2578 H=2578:00 +CPU= 25.949
  18:08:37 STEP 2579 H=2579:00 +CPU= 25.374
  18:09:18 STEP 2580 H=2580:00 +CPU= 41.651
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002580
  18:09:45 STEP 2581 H=2581:00 +CPU= 26.192
  18:10:11 STEP 2582 H=2582:00 +CPU= 25.908
  18:10:49 STEP 2583 H=2583:00 +CPU= 38.588
  18:11:15 STEP 2584 H=2584:00 +CPU= 25.617
  18:11:41 STEP 2585 H=2585:00 +CPU= 26.265
  18:12:22 STEP 2586 H=2586:00 +CPU= 40.534
  18:12:48 STEP 2587 H=2587:00 +CPU= 26.574
  18:13:14 STEP 2588 H=2588:00 +CPU= 26.293
  18:13:56 STEP 2589 H=2589:00 +CPU= 41.158
  18:14:18 STEP 2590 H=2590:00 +CPU= 22.606
  18:14:45 STEP 2591 H=2591:00 +CPU= 27.039
  18:15:28 STEP 2592 H=2592:00 +CPU= 42.229
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_107.zip
Uploading the intermediate file: upload_file_107.zip
  18:15:53 STEP 2593 H=2593:00 +CPU= 25.430
  18:16:17 STEP 2594 H=2594:00 +CPU= 23.910
  18:16:52 STEP 2595 H=2595:00 +CPU= 34.689
  18:17:19 STEP 2596 H=2596:00 +CPU= 27.001
  18:17:44 STEP 2597 H=2597:00 +CPU= 25.685
  18:18:23 STEP 2598 H=2598:00 +CPU= 38.700
  18:18:48 STEP 2599 H=2599:00 +CPU= 24.468
  18:19:12 STEP 2600 H=2600:00 +CPU= 24.864
  18:19:51 STEP 2601 H=2601:00 +CPU= 38.207
  18:20:16 STEP 2602 H=2602:00 +CPU= 25.174
  18:20:40 STEP 2603 H=2603:00 +CPU= 23.627
  18:21:20 STEP 2604 H=2604:00 +CPU= 40.155
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002604
  18:21:43 STEP 2605 H=2605:00 +CPU= 23.478
  18:22:06 STEP 2606 H=2606:00 +CPU= 23.233
  18:22:40 STEP 2607 H=2607:00 +CPU= 33.562
  18:23:01 STEP 2608 H=2608:00 +CPU= 21.129
  18:23:25 STEP 2609 H=2609:00 +CPU= 23.822
  18:24:02 STEP 2610 H=2610:00 +CPU= 37.032
  18:24:27 STEP 2611 H=2611:00 +CPU= 25.057
  18:24:52 STEP 2612 H=2612:00 +CPU= 25.361
  18:25:31 STEP 2613 H=2613:00 +CPU= 38.552
  18:25:55 STEP 2614 H=2614:00 +CPU= 23.860
  18:26:22 STEP 2615 H=2615:00 +CPU= 26.868
  18:27:06 STEP 2616 H=2616:00 +CPU= 44.639
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_108.zip
Uploading the intermediate file: upload_file_108.zip
  18:27:27 STEP 2617 H=2617:00 +CPU= 20.188
  18:27:51 STEP 2618 H=2618:00 +CPU= 24.261
  18:28:26 STEP 2619 H=2619:00 +CPU= 35.570
  18:28:53 STEP 2620 H=2620:00 +CPU= 26.089
  18:29:17 STEP 2621 H=2621:00 +CPU= 24.370
  18:29:55 STEP 2622 H=2622:00 +CPU= 38.298
  18:30:19 STEP 2623 H=2623:00 +CPU= 23.897
  18:30:42 STEP 2624 H=2624:00 +CPU= 23.110
  18:31:18 STEP 2625 H=2625:00 +CPU= 35.752
  18:31:40 STEP 2626 H=2626:00 +CPU= 22.017
  18:31:59 STEP 2627 H=2627:00 +CPU= 19.168
  18:32:37 STEP 2628 H=2628:00 +CPU= 37.502
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002628
  18:32:58 STEP 2629 H=2629:00 +CPU= 21.097
  18:33:23 STEP 2630 H=2630:00 +CPU= 25.029
  18:33:59 STEP 2631 H=2631:00 +CPU= 35.667
  18:34:22 STEP 2632 H=2632:00 +CPU= 23.522
  18:34:46 STEP 2633 H=2633:00 +CPU= 23.880
  18:35:25 STEP 2634 H=2634:00 +CPU= 38.909
  18:35:48 STEP 2635 H=2635:00 +CPU= 23.146
  18:36:12 STEP 2636 H=2636:00 +CPU= 24.077
  18:36:48 STEP 2637 H=2637:00 +CPU= 35.836
  18:37:12 STEP 2638 H=2638:00 +CPU= 23.886
  18:37:37 STEP 2639 H=2639:00 +CPU= 24.514
  18:38:25 STEP 2640 H=2640:00 +CPU= 48.122
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_109.zip
Uploading the intermediate file: upload_file_109.zip
  18:38:50 STEP 2641 H=2641:00 +CPU= 25.477
Uploading trickle at timestep: 9504000
  18:39:16 STEP 2642 H=2642:00 +CPU= 25.436
  18:39:55 STEP 2643 H=2643:00 +CPU= 38.939
  18:40:22 STEP 2644 H=2644:00 +CPU= 26.714
  18:40:48 STEP 2645 H=2645:00 +CPU= 26.527
  18:41:27 STEP 2646 H=2646:00 +CPU= 38.880
  18:41:52 STEP 2647 H=2647:00 +CPU= 25.051
  18:42:16 STEP 2648 H=2648:00 +CPU= 23.973
  18:42:49 STEP 2649 H=2649:00 +CPU= 33.330
  18:43:13 STEP 2650 H=2650:00 +CPU= 23.324
  18:43:37 STEP 2651 H=2651:00 +CPU= 24.374
  18:44:20 STEP 2652 H=2652:00 +CPU= 42.759
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002652
  18:44:44 STEP 2653 H=2653:00 +CPU= 24.260
  18:45:09 STEP 2654 H=2654:00 +CPU= 24.309
  18:45:48 STEP 2655 H=2655:00 +CPU= 38.813
  18:46:13 STEP 2656 H=2656:00 +CPU= 24.996
  18:46:36 STEP 2657 H=2657:00 +CPU= 23.448
  18:47:13 STEP 2658 H=2658:00 +CPU= 37.130
  18:47:37 STEP 2659 H=2659:00 +CPU= 24.031
  18:47:59 STEP 2660 H=2660:00 +CPU= 21.601
  18:48:35 STEP 2661 H=2661:00 +CPU= 36.145
  18:49:00 STEP 2662 H=2662:00 +CPU= 25.260
  18:49:24 STEP 2663 H=2663:00 +CPU= 24.191
  18:50:07 STEP 2664 H=2664:00 +CPU= 42.628
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_110.zip
Uploading the intermediate file: upload_file_110.zip
  18:50:31 STEP 2665 H=2665:00 +CPU= 24.057
  18:50:54 STEP 2666 H=2666:00 +CPU= 22.413
  18:51:27 STEP 2667 H=2667:00 +CPU= 32.974
  18:51:52 STEP 2668 H=2668:00 +CPU= 25.090
  18:52:16 STEP 2669 H=2669:00 +CPU= 24.082
  18:52:54 STEP 2670 H=2670:00 +CPU= 38.061
  18:53:20 STEP 2671 H=2671:00 +CPU= 25.983
  18:53:47 STEP 2672 H=2672:00 +CPU= 26.802
  18:54:27 STEP 2673 H=2673:00 +CPU= 40.296
  18:54:53 STEP 2674 H=2674:00 +CPU= 25.758
  18:55:17 STEP 2675 H=2675:00 +CPU= 24.758
  18:55:57 STEP 2676 H=2676:00 +CPU= 39.073
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002676
  18:56:21 STEP 2677 H=2677:00 +CPU= 24.819
  18:56:45 STEP 2678 H=2678:00 +CPU= 23.932
  18:57:22 STEP 2679 H=2679:00 +CPU= 36.468
  18:57:46 STEP 2680 H=2680:00 +CPU= 24.512
  18:58:11 STEP 2681 H=2681:00 +CPU= 24.465
  18:58:51 STEP 2682 H=2682:00 +CPU= 40.113
  18:59:16 STEP 2683 H=2683:00 +CPU= 25.567
  18:59:44 STEP 2684 H=2684:00 +CPU= 27.156
  19:00:21 STEP 2685 H=2685:00 +CPU= 37.615
  19:00:44 STEP 2686 H=2686:00 +CPU= 22.762
  19:01:08 STEP 2687 H=2687:00 +CPU= 24.358
  19:01:50 STEP 2688 H=2688:00 +CPU= 41.283
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_111.zip
Uploading the intermediate file: upload_file_111.zip
  19:02:13 STEP 2689 H=2689:00 +CPU= 22.856
  19:02:35 STEP 2690 H=2690:00 +CPU= 21.998
  19:03:13 STEP 2691 H=2691:00 +CPU= 38.821
  19:03:40 STEP 2692 H=2692:00 +CPU= 26.642
  19:04:05 STEP 2693 H=2693:00 +CPU= 25.195
  19:04:44 STEP 2694 H=2694:00 +CPU= 38.892
  19:05:10 STEP 2695 H=2695:00 +CPU= 26.232
  19:05:36 STEP 2696 H=2696:00 +CPU= 25.920
  19:06:12 STEP 2697 H=2697:00 +CPU= 36.168
  19:06:33 STEP 2698 H=2698:00 +CPU= 20.257
  19:06:54 STEP 2699 H=2699:00 +CPU= 21.700
  19:07:32 STEP 2700 H=2700:00 +CPU= 37.558
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002700
  19:07:54 STEP 2701 H=2701:00 +CPU= 22.408
  19:08:19 STEP 2702 H=2702:00 +CPU= 24.539
  19:08:54 STEP 2703 H=2703:00 +CPU= 35.207
  19:09:18 STEP 2704 H=2704:00 +CPU= 24.077
  19:09:42 STEP 2705 H=2705:00 +CPU= 23.764
  19:10:20 STEP 2706 H=2706:00 +CPU= 38.065
  19:10:42 STEP 2707 H=2707:00 +CPU= 22.389
  19:11:07 STEP 2708 H=2708:00 +CPU= 24.568
  19:11:42 STEP 2709 H=2709:00 +CPU= 35.423
  19:12:07 STEP 2710 H=2710:00 +CPU= 25.001
  19:12:31 STEP 2711 H=2711:00 +CPU= 23.252
  19:13:15 STEP 2712 H=2712:00 +CPU= 44.620
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_112.zip
Uploading the intermediate file: upload_file_112.zip
  19:13:38 STEP 2713 H=2713:00 +CPU= 22.834
  19:14:02 STEP 2714 H=2714:00 +CPU= 24.154
  19:14:40 STEP 2715 H=2715:00 +CPU= 37.107
  19:15:03 STEP 2716 H=2716:00 +CPU= 23.123
  19:15:25 STEP 2717 H=2717:00 +CPU= 22.766
  19:16:03 STEP 2718 H=2718:00 +CPU= 37.491
  19:16:26 STEP 2719 H=2719:00 +CPU= 23.414
  19:16:51 STEP 2720 H=2720:00 +CPU= 24.181
  19:17:25 STEP 2721 H=2721:00 +CPU= 33.933
  19:17:54 STEP 2722 H=2722:00 +CPU= 29.642
  19:18:23 STEP 2723 H=2723:00 +CPU= 29.163
  19:19:08 STEP 2724 H=2724:00 +CPU= 44.261
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002724
  19:19:30 STEP 2725 H=2725:00 +CPU= 22.438
  19:19:56 STEP 2726 H=2726:00 +CPU= 25.514
  19:20:28 STEP 2727 H=2727:00 +CPU= 31.973
  19:20:50 STEP 2728 H=2728:00 +CPU= 21.989
  19:21:09 STEP 2729 H=2729:00 +CPU= 19.711
  19:21:45 STEP 2730 H=2730:00 +CPU= 35.784
  19:22:05 STEP 2731 H=2731:00 +CPU= 20.360
  19:22:28 STEP 2732 H=2732:00 +CPU= 22.545
  19:23:02 STEP 2733 H=2733:00 +CPU= 34.257
  19:23:26 STEP 2734 H=2734:00 +CPU= 23.688
  19:23:49 STEP 2735 H=2735:00 +CPU= 23.415
  19:24:31 STEP 2736 H=2736:00 +CPU= 41.652
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_113.zip
Uploading the intermediate file: upload_file_113.zip
  19:24:54 STEP 2737 H=2737:00 +CPU= 22.661
  19:25:19 STEP 2738 H=2738:00 +CPU= 24.827
  19:25:53 STEP 2739 H=2739:00 +CPU= 33.803
  19:26:15 STEP 2740 H=2740:00 +CPU= 22.855
  19:26:39 STEP 2741 H=2741:00 +CPU= 23.902
  19:27:16 STEP 2742 H=2742:00 +CPU= 37.148
  19:27:40 STEP 2743 H=2743:00 +CPU= 23.674
  19:28:05 STEP 2744 H=2744:00 +CPU= 24.313
  19:28:39 STEP 2745 H=2745:00 +CPU= 34.040
  19:29:02 STEP 2746 H=2746:00 +CPU= 23.446
  19:29:25 STEP 2747 H=2747:00 +CPU= 23.393
  19:30:08 STEP 2748 H=2748:00 +CPU= 42.709
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002748
  19:30:31 STEP 2749 H=2749:00 +CPU= 22.577
  19:31:01 STEP 2750 H=2750:00 +CPU= 29.823
  19:31:38 STEP 2751 H=2751:00 +CPU= 37.880
  19:32:02 STEP 2752 H=2752:00 +CPU= 23.361
  19:32:26 STEP 2753 H=2753:00 +CPU= 24.018
  19:33:07 STEP 2754 H=2754:00 +CPU= 41.178
  19:33:32 STEP 2755 H=2755:00 +CPU= 24.652
  19:33:57 STEP 2756 H=2756:00 +CPU= 25.018
  19:34:33 STEP 2757 H=2757:00 +CPU= 36.466
  19:34:58 STEP 2758 H=2758:00 +CPU= 24.716
  19:35:21 STEP 2759 H=2759:00 +CPU= 23.528
  19:36:04 STEP 2760 H=2760:00 +CPU= 42.024
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_114.zip
Uploading the intermediate file: upload_file_114.zip
  19:36:26 STEP 2761 H=2761:00 +CPU= 22.701
  19:36:54 STEP 2762 H=2762:00 +CPU= 27.501
  19:37:27 STEP 2763 H=2763:00 +CPU= 33.363
  19:37:50 STEP 2764 H=2764:00 +CPU= 23.057
  19:38:14 STEP 2765 H=2765:00 +CPU= 23.477
  19:38:52 STEP 2766 H=2766:00 +CPU= 38.342
  19:39:13 STEP 2767 H=2767:00 +CPU= 21.252
  19:39:37 STEP 2768 H=2768:00 +CPU= 23.097
  19:40:10 STEP 2769 H=2769:00 +CPU= 33.973
  19:40:33 STEP 2770 H=2770:00 +CPU= 22.846
  19:40:55 STEP 2771 H=2771:00 +CPU= 21.268
  19:41:37 STEP 2772 H=2772:00 +CPU= 42.356
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002772
  19:41:59 STEP 2773 H=2773:00 +CPU= 21.881
  19:42:23 STEP 2774 H=2774:00 +CPU= 24.594
  19:43:08 STEP 2775 H=2775:00 +CPU= 44.195
  19:43:32 STEP 2776 H=2776:00 +CPU= 23.987
  19:43:55 STEP 2777 H=2777:00 +CPU= 23.128
  19:44:31 STEP 2778 H=2778:00 +CPU= 36.614
  19:44:55 STEP 2779 H=2779:00 +CPU= 23.920
  19:45:19 STEP 2780 H=2780:00 +CPU= 24.069
  19:45:54 STEP 2781 H=2781:00 +CPU= 34.226
  19:46:15 STEP 2782 H=2782:00 +CPU= 21.556
  19:46:38 STEP 2783 H=2783:00 +CPU= 22.730
  19:47:17 STEP 2784 H=2784:00 +CPU= 38.566
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_115.zip
Uploading the intermediate file: upload_file_115.zip
  19:47:36 STEP 2785 H=2785:00 +CPU= 19.862
  19:47:57 STEP 2786 H=2786:00 +CPU= 20.373
  19:48:28 STEP 2787 H=2787:00 +CPU= 31.265
  19:48:50 STEP 2788 H=2788:00 +CPU= 22.188
  19:49:11 STEP 2789 H=2789:00 +CPU= 21.056
  19:49:46 STEP 2790 H=2790:00 +CPU= 34.283
  19:50:04 STEP 2791 H=2791:00 +CPU= 18.659
  19:50:26 STEP 2792 H=2792:00 +CPU= 21.818
  19:51:00 STEP 2793 H=2793:00 +CPU= 33.752
  19:51:23 STEP 2794 H=2794:00 +CPU= 23.154
  19:51:46 STEP 2795 H=2795:00 +CPU= 22.955
  19:52:25 STEP 2796 H=2796:00 +CPU= 38.729
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002796
  19:52:48 STEP 2797 H=2797:00 +CPU= 22.705
  19:53:10 STEP 2798 H=2798:00 +CPU= 22.610
  19:53:44 STEP 2799 H=2799:00 +CPU= 33.610
  19:54:07 STEP 2800 H=2800:00 +CPU= 23.167
  19:54:30 STEP 2801 H=2801:00 +CPU= 22.405
  19:55:07 STEP 2802 H=2802:00 +CPU= 37.553
  19:55:31 STEP 2803 H=2803:00 +CPU= 24.310
  19:55:55 STEP 2804 H=2804:00 +CPU= 23.520
  19:56:29 STEP 2805 H=2805:00 +CPU= 34.044
  19:56:54 STEP 2806 H=2806:00 +CPU= 25.388
  19:57:14 STEP 2807 H=2807:00 +CPU= 20.001
  19:57:55 STEP 2808 H=2808:00 +CPU= 40.293
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_116.zip
Uploading the intermediate file: upload_file_116.zip
  19:58:17 STEP 2809 H=2809:00 +CPU= 22.317
  19:58:41 STEP 2810 H=2810:00 +CPU= 23.494
  19:59:16 STEP 2811 H=2811:00 +CPU= 35.295
  19:59:38 STEP 2812 H=2812:00 +CPU= 22.520
  20:00:02 STEP 2813 H=2813:00 +CPU= 23.465
  20:00:38 STEP 2814 H=2814:00 +CPU= 36.517
  20:01:01 STEP 2815 H=2815:00 +CPU= 22.680
  20:01:24 STEP 2816 H=2816:00 +CPU= 22.770
  20:02:00 STEP 2817 H=2817:00 +CPU= 35.654
  20:02:23 STEP 2818 H=2818:00 +CPU= 23.396
  20:02:45 STEP 2819 H=2819:00 +CPU= 22.281
  20:03:26 STEP 2820 H=2820:00 +CPU= 40.391
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002820
  20:03:49 STEP 2821 H=2821:00 +CPU= 23.477
  20:04:12 STEP 2822 H=2822:00 +CPU= 22.644
  20:04:50 STEP 2823 H=2823:00 +CPU= 38.375
  20:05:14 STEP 2824 H=2824:00 +CPU= 24.076
  20:05:33 STEP 2825 H=2825:00 +CPU= 19.042
  20:06:09 STEP 2826 H=2826:00 +CPU= 35.124
  20:06:32 STEP 2827 H=2827:00 +CPU= 23.757
  20:06:56 STEP 2828 H=2828:00 +CPU= 23.220
  20:07:33 STEP 2829 H=2829:00 +CPU= 37.202
  20:07:58 STEP 2830 H=2830:00 +CPU= 24.987
  20:08:23 STEP 2831 H=2831:00 +CPU= 24.743
  20:09:08 STEP 2832 H=2832:00 +CPU= 45.604
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_117.zip
Uploading the intermediate file: upload_file_117.zip
  20:09:28 STEP 2833 H=2833:00 +CPU= 19.776
  20:09:51 STEP 2834 H=2834:00 +CPU= 22.987
  20:10:27 STEP 2835 H=2835:00 +CPU= 36.125
  20:10:52 STEP 2836 H=2836:00 +CPU= 25.237
  20:11:13 STEP 2837 H=2837:00 +CPU= 20.854
  20:11:48 STEP 2838 H=2838:00 +CPU= 35.192
  20:12:10 STEP 2839 H=2839:00 +CPU= 21.935
  20:12:35 STEP 2840 H=2840:00 +CPU= 24.886
  20:13:13 STEP 2841 H=2841:00 +CPU= 37.732
  20:13:37 STEP 2842 H=2842:00 +CPU= 24.307
  20:13:59 STEP 2843 H=2843:00 +CPU= 21.767
  20:14:38 STEP 2844 H=2844:00 +CPU= 38.586
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002844
  20:15:02 STEP 2845 H=2845:00 +CPU= 23.953
  20:15:24 STEP 2846 H=2846:00 +CPU= 22.389
  20:15:59 STEP 2847 H=2847:00 +CPU= 34.931
  20:16:22 STEP 2848 H=2848:00 +CPU= 23.225
  20:16:44 STEP 2849 H=2849:00 +CPU= 21.908
  20:17:20 STEP 2850 H=2850:00 +CPU= 35.753
  20:17:42 STEP 2851 H=2851:00 +CPU= 21.985
  20:18:05 STEP 2852 H=2852:00 +CPU= 22.698
  20:18:40 STEP 2853 H=2853:00 +CPU= 34.944
  20:19:03 STEP 2854 H=2854:00 +CPU= 23.394
  20:19:27 STEP 2855 H=2855:00 +CPU= 23.914
  20:20:09 STEP 2856 H=2856:00 +CPU= 42.090
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_118.zip
Uploading the intermediate file: upload_file_118.zip
  20:20:31 STEP 2857 H=2857:00 +CPU= 22.308
  20:20:53 STEP 2858 H=2858:00 +CPU= 21.229
  20:21:24 STEP 2859 H=2859:00 +CPU= 31.021
  20:21:47 STEP 2860 H=2860:00 +CPU= 23.436
  20:22:10 STEP 2861 H=2861:00 +CPU= 22.438
  20:22:45 STEP 2862 H=2862:00 +CPU= 35.358
  20:23:08 STEP 2863 H=2863:00 +CPU= 22.956
  20:23:31 STEP 2864 H=2864:00 +CPU= 23.331
  20:24:07 STEP 2865 H=2865:00 +CPU= 36.111
  20:24:32 STEP 2866 H=2866:00 +CPU= 24.584
  20:24:54 STEP 2867 H=2867:00 +CPU= 22.384
  20:25:33 STEP 2868 H=2868:00 +CPU= 38.795
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002868
  20:25:56 STEP 2869 H=2869:00 +CPU= 22.823
  20:26:16 STEP 2870 H=2870:00 +CPU= 19.718
  20:26:50 STEP 2871 H=2871:00 +CPU= 34.233
  20:27:12 STEP 2872 H=2872:00 +CPU= 22.083
  20:27:35 STEP 2873 H=2873:00 +CPU= 22.725
  20:28:11 STEP 2874 H=2874:00 +CPU= 36.365
  20:28:34 STEP 2875 H=2875:00 +CPU= 22.934
  20:28:57 STEP 2876 H=2876:00 +CPU= 23.278
  20:29:31 STEP 2877 H=2877:00 +CPU= 34.190
  20:29:55 STEP 2878 H=2878:00 +CPU= 23.036
  20:30:17 STEP 2879 H=2879:00 +CPU= 22.504
  20:30:59 STEP 2880 H=2880:00 +CPU= 41.827
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_119.zip
Uploading the intermediate file: upload_file_119.zip
  20:31:22 STEP 2881 H=2881:00 +CPU= 23.333
Uploading trickle at timestep: 10368000
  20:31:47 STEP 2882 H=2882:00 +CPU= 24.655
  20:32:23 STEP 2883 H=2883:00 +CPU= 35.783
  20:32:46 STEP 2884 H=2884:00 +CPU= 22.989
  20:33:07 STEP 2885 H=2885:00 +CPU= 21.303
  20:33:43 STEP 2886 H=2886:00 +CPU= 35.743
  20:34:05 STEP 2887 H=2887:00 +CPU= 22.126
  20:34:28 STEP 2888 H=2888:00 +CPU= 23.266
  20:35:02 STEP 2889 H=2889:00 +CPU= 33.851
  20:35:26 STEP 2890 H=2890:00 +CPU= 23.875
  20:35:47 STEP 2891 H=2891:00 +CPU= 20.882
  20:36:25 STEP 2892 H=2892:00 +CPU= 37.892
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002892
  20:36:47 STEP 2893 H=2893:00 +CPU= 21.798
  20:37:09 STEP 2894 H=2894:00 +CPU= 22.057
  20:37:44 STEP 2895 H=2895:00 +CPU= 35.519
  20:38:08 STEP 2896 H=2896:00 +CPU= 23.471
  20:38:30 STEP 2897 H=2897:00 +CPU= 22.145
  20:39:05 STEP 2898 H=2898:00 +CPU= 34.738
  20:39:26 STEP 2899 H=2899:00 +CPU= 21.601
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0886
(argv4) batchid: 987
(argv5) wuid: 12204530
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/6
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c1ae5803f8ac5e8d5ad4765f59eafa52 to: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_0886_2018050100_123_987_12204530.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_0886_2018050100_123_987_12204530.zip
ic_ancil_file: ic_ancil_12204530
ifsdata_file: ifsdata_12204530
climate_data_file: clim_data_12204530
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b73891d866a923fd582c819e7820bae to: /var/lib/boinc-client_40000/slots/6/ic_ancil_12204530.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/6/ic_ancil_12204530.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d2a8db77595644a27227fe44500431ac to: /var/lib/boinc-client_40000/slots/6/ifsdata/ifsdata_12204530.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/6/ifsdata/ifsdata_12204530.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/6/159l_2/clim_data_12204530.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/6/159l_2/clim_data_12204530.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/6/progress_file_12204530.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/6/progress_file_12204530.xml
-- Model is restarting --
Adjusting last_iter, 2900, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/6/progress_file_12204530.xml
last_cpu_time: 76528
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897
The child process has been launched with process id: 1451
Executing the command: /var/lib/boinc-client_40000/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1451:1451] [20230118:204956:1674074996:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:51:22 STEP 2880 H=2880:00 +CPU= 34.478
  20:51:48 STEP 2881 H=2881:00 +CPU= 25.790
  20:52:15 STEP 2882 H=2882:00 +CPU= 27.393
  20:52:51 STEP 2883 H=2883:00 +CPU= 36.112
  20:53:14 STEP 2884 H=2884:00 +CPU= 22.479
  20:53:40 STEP 2885 H=2885:00 +CPU= 26.450
  20:54:20 STEP 2886 H=2886:00 +CPU= 39.651
  20:54:45 STEP 2887 H=2887:00 +CPU= 25.319
  20:55:12 STEP 2888 H=2888:00 +CPU= 26.875
  20:55:50 STEP 2889 H=2889:00 +CPU= 38.071
  20:56:14 STEP 2890 H=2890:00 +CPU= 23.695
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2018050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0886
(argv4) batchid: 987
(argv5) wuid: 12204530
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/6
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c1ae5803f8ac5e8d5ad4765f59eafa52 to: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_0886_2018050100_123_987_12204530.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/6/oifs_43r3_ps_0886_2018050100_123_987_12204530.zip
ic_ancil_file: ic_ancil_12204530
ifsdata_file: ifsdata_12204530
climate_data_file: clim_data_12204530
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5b73891d866a923fd582c819e7820bae to: /var/lib/boinc-client_40000/slots/6/ic_ancil_12204530.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/6/ic_ancil_12204530.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d2a8db77595644a27227fe44500431ac to: /var/lib/boinc-client_40000/slots/6/ifsdata/ifsdata_12204530.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/6/ifsdata/ifsdata_12204530.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/6/159l_2/clim_data_12204530.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/6/159l_2/clim_data_12204530.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/6/progress_file_12204530.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/6/progress_file_12204530.xml
-- Model is restarting --
Adjusting last_iter, 2891, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/6/progress_file_12204530.xml
last_cpu_time: 76868
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897
The child process has been launched with process id: 1420
Executing the command: /var/lib/boinc-client_40000/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1420:1420] [20230118:210603:1674075963:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:07:57 STEP 2880 H=2880:00 +CPU= 23.990
  21:08:14 STEP 2881 H=2881:00 +CPU= 16.740
  21:08:36 STEP 2882 H=2882:00 +CPU= 22.002
  21:09:11 STEP 2883 H=2883:00 +CPU= 34.975
  21:09:34 STEP 2884 H=2884:00 +CPU= 23.413
  21:09:57 STEP 2885 H=2885:00 +CPU= 22.793
  21:10:36 STEP 2886 H=2886:00 +CPU= 39.322
  21:11:00 STEP 2887 H=2887:00 +CPU= 23.936
  21:11:24 STEP 2888 H=2888:00 +CPU= 23.751
  21:11:58 STEP 2889 H=2889:00 +CPU= 33.600
  21:12:23 STEP 2890 H=2890:00 +CPU= 25.218
  21:12:47 STEP 2891 H=2891:00 +CPU= 24.567
  21:13:27 STEP 2892 H=2892:00 +CPU= 39.968
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002892
  21:13:49 STEP 2893 H=2893:00 +CPU= 22.008
  21:14:12 STEP 2894 H=2894:00 +CPU= 22.246
  21:14:45 STEP 2895 H=2895:00 +CPU= 33.450
  21:15:07 STEP 2896 H=2896:00 +CPU= 21.907
  21:15:29 STEP 2897 H=2897:00 +CPU= 21.971
  21:16:05 STEP 2898 H=2898:00 +CPU= 35.973
  21:16:27 STEP 2899 H=2899:00 +CPU= 21.610
  21:16:46 STEP 2900 H=2900:00 +CPU= 19.539
  21:17:18 STEP 2901 H=2901:00 +CPU= 32.017
  21:17:42 STEP 2902 H=2902:00 +CPU= 23.651
  21:18:05 STEP 2903 H=2903:00 +CPU= 22.856
  21:18:46 STEP 2904 H=2904:00 +CPU= 41.446
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_120.zip
Uploading the intermediate file: upload_file_120.zip
  21:19:10 STEP 2905 H=2905:00 +CPU= 23.379
  21:19:33 STEP 2906 H=2906:00 +CPU= 22.923
  21:20:09 STEP 2907 H=2907:00 +CPU= 35.861
  21:20:33 STEP 2908 H=2908:00 +CPU= 23.609
  21:20:55 STEP 2909 H=2909:00 +CPU= 22.369
  21:21:31 STEP 2910 H=2910:00 +CPU= 36.176
  21:21:55 STEP 2911 H=2911:00 +CPU= 23.521
  21:22:20 STEP 2912 H=2912:00 +CPU= 24.846
  21:22:54 STEP 2913 H=2913:00 +CPU= 34.430
  21:23:19 STEP 2914 H=2914:00 +CPU= 24.618
  21:23:43 STEP 2915 H=2915:00 +CPU= 24.395
  21:24:27 STEP 2916 H=2916:00 +CPU= 43.654
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002916
  21:24:54 STEP 2917 H=2917:00 +CPU= 27.380
  21:25:21 STEP 2918 H=2918:00 +CPU= 27.334
  21:26:02 STEP 2919 H=2919:00 +CPU= 40.428
  21:26:31 STEP 2920 H=2920:00 +CPU= 29.147
  21:26:59 STEP 2921 H=2921:00 +CPU= 27.605
  21:27:41 STEP 2922 H=2922:00 +CPU= 42.659
  21:28:08 STEP 2923 H=2923:00 +CPU= 26.938
  21:28:39 STEP 2924 H=2924:00 +CPU= 30.942
  21:29:20 STEP 2925 H=2925:00 +CPU= 40.583
  21:29:50 STEP 2926 H=2926:00 +CPU= 30.217
  21:30:20 STEP 2927 H=2927:00 +CPU= 29.534
  21:31:07 STEP 2928 H=2928:00 +CPU= 46.971
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client_40000/slots/6/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12204530/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0886_2018050100_123_987_12204530_2_r1171924897_121.zip
Uploading the intermediate file: upload_file_121.zip
  21:31:33 STEP 2929 H=2929:00 +CPU= 25.989

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:27:05 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 10,368,000 76,005 0.0073
09 May 2023 16:26:59 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 9,504,000 69,257 0.0073
09 May 2023 16:26:48 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 8,640,000 62,168 0.0072
09 May 2023 16:26:36 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 7,776,000 55,060 0.0071
09 May 2023 16:26:25 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 6,912,000 48,564 0.0070
09 May 2023 16:26:13 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 6,048,000 42,268 0.0070
09 May 2023 16:26:02 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 5,184,000 36,072 0.0070
09 May 2023 16:25:56 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 4,320,000 29,859 0.0069
09 May 2023 16:25:44 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 3,456,000 23,624 0.0068
09 May 2023 16:25:32 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 2,592,000 17,159 0.0066
09 May 2023 16:25:22 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 1,728,000 10,464 0.0061
09 May 2023 16:25:16 1538236 22298306 oifs_43r3_ps_0886_2018050100_123_987_12204530_2 864,000 4,392 0.0051


©2024 cpdn.org