climateprediction.net home page
Task 22287212

Task 22287212

Name oifs_43r3_ps_0298_2015050100_123_984_12200942_0
Workunit 12200942
Created 24 Dec 2022, 12:25:35 UTC
Sent 19 Jan 2023, 13:03:55 UTC
Report deadline 20 Mar 2023, 13:03:55 UTC
Received 20 Jan 2023, 13:56:02 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 19 hours 30 min 13 sec
CPU time 17 hours 34 min 41 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,401.99 MB
Peak swap size 4,978.08 MB
Peak disk usage 6,077.47 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
:123515:123515] [20230120:030451:1674209091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:123515:123515] [20230120:030451:1674209091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:123515:123515] [20230120:030451:1674209091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:123515:123515] [20230120:030451:1674209091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:123515:123515] [20230120:030451:1674209091:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:123515:123515] [20230120:030451:1674209091:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:05:28 STEP 2544 H=2544:00 +CPU= 32.791
  03:05:47 STEP 2545 H=2545:00 +CPU= 19.420
  03:06:07 STEP 2546 H=2546:00 +CPU= 19.338
  03:06:35 STEP 2547 H=2547:00 +CPU= 27.038
  03:06:51 STEP 2548 H=2548:00 +CPU= 16.070
  03:07:07 STEP 2549 H=2549:00 +CPU= 16.108
  03:07:38 STEP 2550 H=2550:00 +CPU= 30.457
  03:07:59 STEP 2551 H=2551:00 +CPU= 19.717
  03:08:15 STEP 2552 H=2552:00 +CPU= 16.503
  03:08:45 STEP 2553 H=2553:00 +CPU= 29.257
  03:09:06 STEP 2554 H=2554:00 +CPU= 20.470
  03:09:26 STEP 2555 H=2555:00 +CPU= 19.254
  03:09:58 STEP 2556 H=2556:00 +CPU= 32.032
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002556
  03:10:17 STEP 2557 H=2557:00 +CPU= 18.421
  03:10:36 STEP 2558 H=2558:00 +CPU= 18.565
  03:11:04 STEP 2559 H=2559:00 +CPU= 27.092
  03:11:25 STEP 2560 H=2560:00 +CPU= 20.428
  03:11:44 STEP 2561 H=2561:00 +CPU= 18.739
  03:12:15 STEP 2562 H=2562:00 +CPU= 30.142
  03:12:34 STEP 2563 H=2563:00 +CPU= 18.987
  03:12:53 STEP 2564 H=2564:00 +CPU= 18.612
  03:13:22 STEP 2565 H=2565:00 +CPU= 28.203
  03:13:40 STEP 2566 H=2566:00 +CPU= 17.296
  03:13:59 STEP 2567 H=2567:00 +CPU= 18.421
  03:14:33 STEP 2568 H=2568:00 +CPU= 33.449
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_106.zip
Uploading the intermediate file: upload_file_106.zip
  03:14:51 STEP 2569 H=2569:00 +CPU= 17.863
  03:15:11 STEP 2570 H=2570:00 +CPU= 19.194
  03:15:41 STEP 2571 H=2571:00 +CPU= 29.439
  03:16:01 STEP 2572 H=2572:00 +CPU= 19.252
  03:16:19 STEP 2573 H=2573:00 +CPU= 18.223
  03:16:50 STEP 2574 H=2574:00 +CPU= 30.434
  03:17:09 STEP 2575 H=2575:00 +CPU= 18.449
  03:17:29 STEP 2576 H=2576:00 +CPU= 18.980
  03:17:56 STEP 2577 H=2577:00 +CPU= 26.862
  03:18:14 STEP 2578 H=2578:00 +CPU= 18.091
  03:18:33 STEP 2579 H=2579:00 +CPU= 18.302
  03:19:07 STEP 2580 H=2580:00 +CPU= 32.758
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  03:19:26 STEP 2581 H=2581:00 +CPU= 18.935
  03:19:46 STEP 2582 H=2582:00 +CPU= 19.700
  03:20:16 STEP 2583 H=2583:00 +CPU= 29.443
  03:20:36 STEP 2584 H=2584:00 +CPU= 19.308
  03:20:55 STEP 2585 H=2585:00 +CPU= 18.179
  03:21:23 STEP 2586 H=2586:00 +CPU= 27.645
  03:21:40 STEP 2587 H=2587:00 +CPU= 17.025
  03:21:59 STEP 2588 H=2588:00 +CPU= 18.217
  03:22:30 STEP 2589 H=2589:00 +CPU= 30.094
  03:22:49 STEP 2590 H=2590:00 +CPU= 18.696
  03:23:08 STEP 2591 H=2591:00 +CPU= 18.465
  03:23:45 STEP 2592 H=2592:00 +CPU= 35.703
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_107.zip
Uploading the intermediate file: upload_file_107.zip
  03:24:04 STEP 2593 H=2593:00 +CPU= 18.856
  03:24:21 STEP 2594 H=2594:00 +CPU= 17.241
  03:24:51 STEP 2595 H=2595:00 +CPU= 28.943
  03:25:11 STEP 2596 H=2596:00 +CPU= 20.030
  03:25:31 STEP 2597 H=2597:00 +CPU= 18.935
  03:25:58 STEP 2598 H=2598:00 +CPU= 27.278
  03:26:17 STEP 2599 H=2599:00 +CPU= 18.494
  03:26:36 STEP 2600 H=2600:00 +CPU= 18.004
  03:27:05 STEP 2601 H=2601:00 +CPU= 29.058
  03:27:25 STEP 2602 H=2602:00 +CPU= 19.152
  03:27:44 STEP 2603 H=2603:00 +CPU= 18.615
  03:28:19 STEP 2604 H=2604:00 +CPU= 34.291
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002604
  03:28:40 STEP 2605 H=2605:00 +CPU= 20.043
  03:29:00 STEP 2606 H=2606:00 +CPU= 19.326
  03:29:30 STEP 2607 H=2607:00 +CPU= 30.107
  03:29:50 STEP 2608 H=2608:00 +CPU= 19.558
  03:30:09 STEP 2609 H=2609:00 +CPU= 18.899
  03:30:40 STEP 2610 H=2610:00 +CPU= 30.084
  03:30:59 STEP 2611 H=2611:00 +CPU= 17.980
  03:31:17 STEP 2612 H=2612:00 +CPU= 17.933
  03:31:45 STEP 2613 H=2613:00 +CPU= 27.412
  03:32:04 STEP 2614 H=2614:00 +CPU= 18.543
  03:32:24 STEP 2615 H=2615:00 +CPU= 19.234
  03:32:59 STEP 2616 H=2616:00 +CPU= 34.557
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_108.zip
Uploading the intermediate file: upload_file_108.zip
  03:33:19 STEP 2617 H=2617:00 +CPU= 19.537
  03:33:37 STEP 2618 H=2618:00 +CPU= 17.769
  03:34:08 STEP 2619 H=2619:00 +CPU= 29.435
  03:34:27 STEP 2620 H=2620:00 +CPU= 19.503
  03:34:46 STEP 2621 H=2621:00 +CPU= 18.239
  03:35:20 STEP 2622 H=2622:00 +CPU= 32.732
  03:35:38 STEP 2623 H=2623:00 +CPU= 18.189
  03:35:58 STEP 2624 H=2624:00 +CPU= 19.300
  03:36:25 STEP 2625 H=2625:00 +CPU= 26.892
  03:36:44 STEP 2626 H=2626:00 +CPU= 18.665
  03:37:04 STEP 2627 H=2627:00 +CPU= 18.715
  03:37:36 STEP 2628 H=2628:00 +CPU= 31.592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002628
  03:37:55 STEP 2629 H=2629:00 +CPU= 18.777
  03:38:13 STEP 2630 H=2630:00 +CPU= 17.416
  03:38:41 STEP 2631 H=2631:00 +CPU= 27.941
  03:39:02 STEP 2632 H=2632:00 +CPU= 20.457
  03:39:22 STEP 2633 H=2633:00 +CPU= 19.798
  03:39:55 STEP 2634 H=2634:00 +CPU= 31.651
  03:40:15 STEP 2635 H=2635:00 +CPU= 19.582
  03:40:34 STEP 2636 H=2636:00 +CPU= 18.911
  03:41:03 STEP 2637 H=2637:00 +CPU= 28.115
  03:41:24 STEP 2638 H=2638:00 +CPU= 20.634
  03:41:44 STEP 2639 H=2639:00 +CPU= 19.488
  03:42:18 STEP 2640 H=2640:00 +CPU= 32.810
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_109.zip
Uploading the intermediate file: upload_file_109.zip
  03:42:36 STEP 2641 H=2641:00 +CPU= 18.022
  03:42:54 STEP 2642 H=2642:00 +CPU= 17.775
  03:43:21 STEP 2643 H=2643:00 +CPU= 26.382
  03:43:40 STEP 2644 H=2644:00 +CPU= 18.769
  03:43:59 STEP 2645 H=2645:00 +CPU= 18.701
  03:44:28 STEP 2646 H=2646:00 +CPU= 28.475
  03:44:46 STEP 2647 H=2647:00 +CPU= 18.027
  03:45:06 STEP 2648 H=2648:00 +CPU= 19.168
  03:45:34 STEP 2649 H=2649:00 +CPU= 27.959
  03:45:54 STEP 2650 H=2650:00 +CPU= 18.959
  03:46:13 STEP 2651 H=2651:00 +CPU= 18.690
  03:46:45 STEP 2652 H=2652:00 +CPU= 31.070
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002652
  03:47:05 STEP 2653 H=2653:00 +CPU= 19.542
  03:47:23 STEP 2654 H=2654:00 +CPU= 18.209
  03:47:50 STEP 2655 H=2655:00 +CPU= 26.311
  03:48:08 STEP 2656 H=2656:00 +CPU= 18.280
  03:48:28 STEP 2657 H=2657:00 +CPU= 18.874
  03:48:57 STEP 2658 H=2658:00 +CPU= 28.984
  03:49:16 STEP 2659 H=2659:00 +CPU= 17.982
  03:49:33 STEP 2660 H=2660:00 +CPU= 17.363
  03:50:01 STEP 2661 H=2661:00 +CPU= 27.453
  03:50:22 STEP 2662 H=2662:00 +CPU= 20.661
  03:50:43 STEP 2663 H=2663:00 +CPU= 19.855
  03:51:17 STEP 2664 H=2664:00 +CPU= 32.797
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_110.zip
Uploading the intermediate file: upload_file_110.zip
  03:51:37 STEP 2665 H=2665:00 +CPU= 19.766
  03:51:56 STEP 2666 H=2666:00 +CPU= 18.852
  03:52:22 STEP 2667 H=2667:00 +CPU= 25.648
  03:52:40 STEP 2668 H=2668:00 +CPU= 17.785
  03:52:59 STEP 2669 H=2669:00 +CPU= 18.494
  03:53:29 STEP 2670 H=2670:00 +CPU= 29.615
  03:53:47 STEP 2671 H=2671:00 +CPU= 17.322
  03:54:05 STEP 2672 H=2672:00 +CPU= 17.571
  03:54:33 STEP 2673 H=2673:00 +CPU= 27.868
  03:54:53 STEP 2674 H=2674:00 +CPU= 19.204
  03:55:10 STEP 2675 H=2675:00 +CPU= 17.041
  03:55:40 STEP 2676 H=2676:00 +CPU= 29.107
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002676
  03:55:59 STEP 2677 H=2677:00 +CPU= 18.606
  03:56:17 STEP 2678 H=2678:00 +CPU= 17.898
  03:56:45 STEP 2679 H=2679:00 +CPU= 27.518
  03:57:04 STEP 2680 H=2680:00 +CPU= 18.930
  03:57:25 STEP 2681 H=2681:00 +CPU= 19.566
  03:57:53 STEP 2682 H=2682:00 +CPU= 27.979
  03:58:10 STEP 2683 H=2683:00 +CPU= 16.517
  03:58:28 STEP 2684 H=2684:00 +CPU= 18.141
  03:58:56 STEP 2685 H=2685:00 +CPU= 27.271
  03:59:16 STEP 2686 H=2686:00 +CPU= 19.846
  03:59:36 STEP 2687 H=2687:00 +CPU= 18.668
  04:00:10 STEP 2688 H=2688:00 +CPU= 33.245
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_111.zip
Uploading the intermediate file: upload_file_111.zip
  04:00:28 STEP 2689 H=2689:00 +CPU= 17.885
  04:00:47 STEP 2690 H=2690:00 +CPU= 18.936
  04:01:15 STEP 2691 H=2691:00 +CPU= 27.102
  04:01:34 STEP 2692 H=2692:00 +CPU= 18.948
  04:01:54 STEP 2693 H=2693:00 +CPU= 19.421
  04:02:22 STEP 2694 H=2694:00 +CPU= 27.499
  04:02:38 STEP 2695 H=2695:00 +CPU= 15.922
  04:02:56 STEP 2696 H=2696:00 +CPU= 17.004
  04:03:26 STEP 2697 H=2697:00 +CPU= 29.062
  04:03:46 STEP 2698 H=2698:00 +CPU= 19.860
  04:04:03 STEP 2699 H=2699:00 +CPU= 16.543
  04:04:34 STEP 2700 H=2700:00 +CPU= 30.528
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002700
  04:04:53 STEP 2701 H=2701:00 +CPU= 19.073
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0298
(argv4) batchid: 984
(argv5) wuid: 12200942
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_75fb67b5d05d29d026dde3f54c2d0bc6 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0298_2015050100_123_984_12200942.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0298_2015050100_123_984_12200942.zip
ic_ancil_file: ic_ancil_12200942
ifsdata_file: ifsdata_12200942
climate_data_file: clim_data_12200942
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_63e32b5aa992df1201d2800a73171fd5 to: /var/lib/boinc-client/slots/1/ic_ancil_12200942.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12200942.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37f37531568aa43d622da0758459d2a3 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200942.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200942.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200942.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200942.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12200942.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12200942.xml
-- Model is restarting --
Adjusting last_iter, 2701, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12200942.xml
last_cpu_time: 57014
upload_file_number: 112
last_iter: 2687
last_upload: 9676800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431
The child process has been launched with process id: 135389
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:135389:135389] [20230120:040558:1674212758:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:06:34 STEP 2688 H=2688:00 +CPU= 32.523
  04:06:51 STEP 2689 H=2689:00 +CPU= 16.524
  04:07:08 STEP 2690 H=2690:00 +CPU= 17.243
  04:07:37 STEP 2691 H=2691:00 +CPU= 28.291
  04:07:57 STEP 2692 H=2692:00 +CPU= 19.851
  04:08:15 STEP 2693 H=2693:00 +CPU= 17.476
  04:08:43 STEP 2694 H=2694:00 +CPU= 27.292
  04:09:02 STEP 2695 H=2695:00 +CPU= 18.572
  04:09:20 STEP 2696 H=2696:00 +CPU= 18.186
  04:09:49 STEP 2697 H=2697:00 +CPU= 28.732
  04:10:09 STEP 2698 H=2698:00 +CPU= 18.692
  04:10:26 STEP 2699 H=2699:00 +CPU= 17.599
  04:10:59 STEP 2700 H=2700:00 +CPU= 31.895
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002700
  04:11:19 STEP 2701 H=2701:00 +CPU= 19.989
  04:11:39 STEP 2702 H=2702:00 +CPU= 18.828
  04:12:07 STEP 2703 H=2703:00 +CPU= 27.517
  04:12:26 STEP 2704 H=2704:00 +CPU= 19.164
  04:12:44 STEP 2705 H=2705:00 +CPU= 17.202
  04:13:13 STEP 2706 H=2706:00 +CPU= 28.853
  04:13:31 STEP 2707 H=2707:00 +CPU= 17.037
  04:13:48 STEP 2708 H=2708:00 +CPU= 16.611
  04:14:16 STEP 2709 H=2709:00 +CPU= 27.467
  04:14:37 STEP 2710 H=2710:00 +CPU= 20.749
  04:14:57 STEP 2711 H=2711:00 +CPU= 19.086
  04:15:31 STEP 2712 H=2712:00 +CPU= 33.346
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_112.zip
Uploading the intermediate file: upload_file_112.zip
  04:15:49 STEP 2713 H=2713:00 +CPU= 17.796
  04:16:09 STEP 2714 H=2714:00 +CPU= 19.211
  04:16:35 STEP 2715 H=2715:00 +CPU= 26.092
  04:16:54 STEP 2716 H=2716:00 +CPU= 18.871
  04:17:13 STEP 2717 H=2717:00 +CPU= 18.468
  04:17:42 STEP 2718 H=2718:00 +CPU= 28.396
  04:18:01 STEP 2719 H=2719:00 +CPU= 18.304
  04:18:21 STEP 2720 H=2720:00 +CPU= 19.855
  04:18:49 STEP 2721 H=2721:00 +CPU= 28.029
  04:19:08 STEP 2722 H=2722:00 +CPU= 18.049
  04:19:26 STEP 2723 H=2723:00 +CPU= 17.522
  04:19:57 STEP 2724 H=2724:00 +CPU= 30.770
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002724
  04:20:16 STEP 2725 H=2725:00 +CPU= 18.045
  04:20:33 STEP 2726 H=2726:00 +CPU= 17.218
  04:21:02 STEP 2727 H=2727:00 +CPU= 27.971
  04:21:22 STEP 2728 H=2728:00 +CPU= 19.929
  04:21:42 STEP 2729 H=2729:00 +CPU= 19.149
  04:22:11 STEP 2730 H=2730:00 +CPU= 28.858
  04:22:31 STEP 2731 H=2731:00 +CPU= 19.168
  04:22:49 STEP 2732 H=2732:00 +CPU= 18.241
  04:23:17 STEP 2733 H=2733:00 +CPU= 26.980
  04:23:36 STEP 2734 H=2734:00 +CPU= 18.615
  04:23:56 STEP 2735 H=2735:00 +CPU= 19.651
  04:24:31 STEP 2736 H=2736:00 +CPU= 33.570
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_113.zip
Uploading the intermediate file: upload_file_113.zip
  04:24:50 STEP 2737 H=2737:00 +CPU= 18.745
  04:25:08 STEP 2738 H=2738:00 +CPU= 18.002
  04:25:35 STEP 2739 H=2739:00 +CPU= 26.337
  04:25:54 STEP 2740 H=2740:00 +CPU= 19.331
  04:26:14 STEP 2741 H=2741:00 +CPU= 18.945
  04:26:43 STEP 2742 H=2742:00 +CPU= 28.456
  04:27:00 STEP 2743 H=2743:00 +CPU= 17.468
  04:27:19 STEP 2744 H=2744:00 +CPU= 18.792
  04:27:49 STEP 2745 H=2745:00 +CPU= 28.640
  04:28:07 STEP 2746 H=2746:00 +CPU= 17.678
  04:28:25 STEP 2747 H=2747:00 +CPU= 17.916
  04:28:57 STEP 2748 H=2748:00 +CPU= 31.565
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002748
  04:29:15 STEP 2749 H=2749:00 +CPU= 17.572
  04:29:33 STEP 2750 H=2750:00 +CPU= 16.748
  04:29:59 STEP 2751 H=2751:00 +CPU= 26.189
  04:30:21 STEP 2752 H=2752:00 +CPU= 21.275
  04:30:39 STEP 2753 H=2753:00 +CPU= 17.308
  04:31:09 STEP 2754 H=2754:00 +CPU= 29.647
  04:31:29 STEP 2755 H=2755:00 +CPU= 19.269
  04:31:46 STEP 2756 H=2756:00 +CPU= 17.377
  04:32:12 STEP 2757 H=2757:00 +CPU= 25.468
  04:32:33 STEP 2758 H=2758:00 +CPU= 20.396
  04:32:55 STEP 2759 H=2759:00 +CPU= 20.565
  04:33:28 STEP 2760 H=2760:00 +CPU= 32.636
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_114.zip
Uploading the intermediate file: upload_file_114.zip
  04:33:46 STEP 2761 H=2761:00 +CPU= 18.114
  04:34:06 STEP 2762 H=2762:00 +CPU= 19.289
  04:34:35 STEP 2763 H=2763:00 +CPU= 27.906
  04:34:53 STEP 2764 H=2764:00 +CPU= 18.361
  04:35:13 STEP 2765 H=2765:00 +CPU= 18.618
  04:35:44 STEP 2766 H=2766:00 +CPU= 31.171
  04:36:05 STEP 2767 H=2767:00 +CPU= 20.464
  04:36:25 STEP 2768 H=2768:00 +CPU= 19.713
  04:36:56 STEP 2769 H=2769:00 +CPU= 29.845
  04:37:16 STEP 2770 H=2770:00 +CPU= 19.908
  04:37:35 STEP 2771 H=2771:00 +CPU= 18.476
  04:38:05 STEP 2772 H=2772:00 +CPU= 29.477
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002772
  04:38:24 STEP 2773 H=2773:00 +CPU= 18.173
  04:38:42 STEP 2774 H=2774:00 +CPU= 17.768
  04:39:11 STEP 2775 H=2775:00 +CPU= 28.191
  04:39:29 STEP 2776 H=2776:00 +CPU= 18.340
  04:39:49 STEP 2777 H=2777:00 +CPU= 18.657
  04:40:17 STEP 2778 H=2778:00 +CPU= 28.045
  04:40:36 STEP 2779 H=2779:00 +CPU= 18.149
  04:40:56 STEP 2780 H=2780:00 +CPU= 19.840
  04:41:25 STEP 2781 H=2781:00 +CPU= 28.505
  04:41:45 STEP 2782 H=2782:00 +CPU= 19.385
  04:42:06 STEP 2783 H=2783:00 +CPU= 20.128
  04:42:41 STEP 2784 H=2784:00 +CPU= 33.761
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_115.zip
Uploading the intermediate file: upload_file_115.zip
  04:43:00 STEP 2785 H=2785:00 +CPU= 18.726
  04:43:20 STEP 2786 H=2786:00 +CPU= 20.050
  04:43:50 STEP 2787 H=2787:00 +CPU= 29.315
  04:44:09 STEP 2788 H=2788:00 +CPU= 19.273
  04:44:29 STEP 2789 H=2789:00 +CPU= 18.976
  04:44:58 STEP 2790 H=2790:00 +CPU= 28.243
  04:45:14 STEP 2791 H=2791:00 +CPU= 16.357
  04:45:32 STEP 2792 H=2792:00 +CPU= 17.477
  04:46:03 STEP 2793 H=2793:00 +CPU= 30.051
  04:46:23 STEP 2794 H=2794:00 +CPU= 19.148
  04:46:39 STEP 2795 H=2795:00 +CPU= 16.040
  04:47:10 STEP 2796 H=2796:00 +CPU= 30.595
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002796
  04:47:30 STEP 2797 H=2797:00 +CPU= 19.318
  04:47:50 STEP 2798 H=2798:00 +CPU= 19.217
  04:48:18 STEP 2799 H=2799:00 +CPU= 27.717
  04:48:38 STEP 2800 H=2800:00 +CPU= 19.710
  04:48:58 STEP 2801 H=2801:00 +CPU= 19.426
  04:49:27 STEP 2802 H=2802:00 +CPU= 28.193
  04:49:43 STEP 2803 H=2803:00 +CPU= 16.229
  04:50:00 STEP 2804 H=2804:00 +CPU= 16.935
  04:50:30 STEP 2805 H=2805:00 +CPU= 29.138
  04:50:51 STEP 2806 H=2806:00 +CPU= 20.211
  04:51:08 STEP 2807 H=2807:00 +CPU= 17.118
  04:51:43 STEP 2808 H=2808:00 +CPU= 33.388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_116.zip
Uploading the intermediate file: upload_file_116.zip
  04:52:01 STEP 2809 H=2809:00 +CPU= 17.693
  04:52:19 STEP 2810 H=2810:00 +CPU= 18.125
  04:52:49 STEP 2811 H=2811:00 +CPU= 28.878
  04:53:10 STEP 2812 H=2812:00 +CPU= 20.143
  04:53:29 STEP 2813 H=2813:00 +CPU= 18.826
  04:53:58 STEP 2814 H=2814:00 +CPU= 28.855
  04:54:16 STEP 2815 H=2815:00 +CPU= 17.345
  04:54:35 STEP 2816 H=2816:00 +CPU= 18.433
  04:55:03 STEP 2817 H=2817:00 +CPU= 27.181
  04:55:25 STEP 2818 H=2818:00 +CPU= 21.189
  04:55:44 STEP 2819 H=2819:00 +CPU= 18.969
  04:56:16 STEP 2820 H=2820:00 +CPU= 30.865
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002820
  04:56:34 STEP 2821 H=2821:00 +CPU= 18.057
  04:56:53 STEP 2822 H=2822:00 +CPU= 18.726
  04:57:21 STEP 2823 H=2823:00 +CPU= 27.307
  04:57:40 STEP 2824 H=2824:00 +CPU= 18.982
  04:58:01 STEP 2825 H=2825:00 +CPU= 19.838
  04:58:32 STEP 2826 H=2826:00 +CPU= 30.386
  04:58:50 STEP 2827 H=2827:00 +CPU= 18.336
  04:59:10 STEP 2828 H=2828:00 +CPU= 18.777
  04:59:37 STEP 2829 H=2829:00 +CPU= 27.241
  04:59:56 STEP 2830 H=2830:00 +CPU= 18.322
  05:00:15 STEP 2831 H=2831:00 +CPU= 18.314
  05:00:49 STEP 2832 H=2832:00 +CPU= 32.992
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_117.zip
Uploading the intermediate file: upload_file_117.zip
  05:01:09 STEP 2833 H=2833:00 +CPU= 19.345
  05:01:28 STEP 2834 H=2834:00 +CPU= 18.864
  05:01:57 STEP 2835 H=2835:00 +CPU= 28.488
  05:02:17 STEP 2836 H=2836:00 +CPU= 19.217
  05:02:34 STEP 2837 H=2837:00 +CPU= 17.226
  05:03:03 STEP 2838 H=2838:00 +CPU= 28.202
  05:03:22 STEP 2839 H=2839:00 +CPU= 18.692
  05:03:41 STEP 2840 H=2840:00 +CPU= 18.629
  05:04:11 STEP 2841 H=2841:00 +CPU= 29.203
  05:04:33 STEP 2842 H=2842:00 +CPU= 21.218
  05:04:53 STEP 2843 H=2843:00 +CPU= 19.779
  05:05:25 STEP 2844 H=2844:00 +CPU= 31.170
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002844
  05:05:43 STEP 2845 H=2845:00 +CPU= 17.766
  05:06:02 STEP 2846 H=2846:00 +CPU= 18.793
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2015050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0298
(argv4) batchid: 984
(argv5) wuid: 12200942
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_75fb67b5d05d29d026dde3f54c2d0bc6 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0298_2015050100_123_984_12200942.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0298_2015050100_123_984_12200942.zip
ic_ancil_file: ic_ancil_12200942
ifsdata_file: ifsdata_12200942
climate_data_file: clim_data_12200942
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_63e32b5aa992df1201d2800a73171fd5 to: /var/lib/boinc-client/slots/1/ic_ancil_12200942.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12200942.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_37f37531568aa43d622da0758459d2a3 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200942.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200942.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200942.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200942.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12200942.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12200942.xml
-- Model is restarting --
Adjusting last_iter, 2847, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12200942.xml
last_cpu_time: 60547
upload_file_number: 118
last_iter: 2831
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431
The child process has been launched with process id: 146119
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:146119:146119] [20230120:050712:1674216432:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:07:46 STEP 2832 H=2832:00 +CPU= 30.545
  05:08:05 STEP 2833 H=2833:00 +CPU= 18.930
  05:08:24 STEP 2834 H=2834:00 +CPU= 18.694
  05:08:52 STEP 2835 H=2835:00 +CPU= 26.985
  05:09:10 STEP 2836 H=2836:00 +CPU= 18.293
  05:09:29 STEP 2837 H=2837:00 +CPU= 17.884
  05:09:59 STEP 2838 H=2838:00 +CPU= 29.880
  05:10:17 STEP 2839 H=2839:00 +CPU= 17.574
  05:10:36 STEP 2840 H=2840:00 +CPU= 18.472
  05:11:07 STEP 2841 H=2841:00 +CPU= 30.285
  05:11:26 STEP 2842 H=2842:00 +CPU= 19.345
  05:11:45 STEP 2843 H=2843:00 +CPU= 18.343
  05:12:16 STEP 2844 H=2844:00 +CPU= 30.218
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002844
  05:12:34 STEP 2845 H=2845:00 +CPU= 17.711
  05:12:53 STEP 2846 H=2846:00 +CPU= 17.987
  05:13:21 STEP 2847 H=2847:00 +CPU= 27.759
  05:13:41 STEP 2848 H=2848:00 +CPU= 19.581
  05:14:00 STEP 2849 H=2849:00 +CPU= 17.957
  05:14:28 STEP 2850 H=2850:00 +CPU= 28.162
  05:14:46 STEP 2851 H=2851:00 +CPU= 17.526
  05:15:05 STEP 2852 H=2852:00 +CPU= 17.771
  05:15:34 STEP 2853 H=2853:00 +CPU= 29.288
  05:15:56 STEP 2854 H=2854:00 +CPU= 20.916
  05:16:16 STEP 2855 H=2855:00 +CPU= 19.753
  05:16:51 STEP 2856 H=2856:00 +CPU= 33.555
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_118.zip
Uploading the intermediate file: upload_file_118.zip
  05:17:10 STEP 2857 H=2857:00 +CPU= 19.063
  05:17:29 STEP 2858 H=2858:00 +CPU= 18.459
  05:17:57 STEP 2859 H=2859:00 +CPU= 27.818
  05:18:18 STEP 2860 H=2860:00 +CPU= 20.125
  05:18:37 STEP 2861 H=2861:00 +CPU= 18.917
  05:19:05 STEP 2862 H=2862:00 +CPU= 26.872
  05:19:24 STEP 2863 H=2863:00 +CPU= 18.758
  05:19:44 STEP 2864 H=2864:00 +CPU= 20.108
  05:20:13 STEP 2865 H=2865:00 +CPU= 28.004
  05:20:31 STEP 2866 H=2866:00 +CPU= 17.789
  05:20:49 STEP 2867 H=2867:00 +CPU= 17.934
  05:21:21 STEP 2868 H=2868:00 +CPU= 30.735
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002868
  05:21:40 STEP 2869 H=2869:00 +CPU= 18.338
  05:21:59 STEP 2870 H=2870:00 +CPU= 18.917
  05:22:28 STEP 2871 H=2871:00 +CPU= 28.053
  05:22:48 STEP 2872 H=2872:00 +CPU= 19.441
  05:23:07 STEP 2873 H=2873:00 +CPU= 18.654
  05:23:37 STEP 2874 H=2874:00 +CPU= 29.534
  05:23:55 STEP 2875 H=2875:00 +CPU= 17.776
  05:24:12 STEP 2876 H=2876:00 +CPU= 17.163
  05:24:42 STEP 2877 H=2877:00 +CPU= 29.068
  05:25:02 STEP 2878 H=2878:00 +CPU= 18.976
  05:25:22 STEP 2879 H=2879:00 +CPU= 19.606
  05:25:55 STEP 2880 H=2880:00 +CPU= 31.945
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_119.zip
Uploading the intermediate file: upload_file_119.zip
  05:26:11 STEP 2881 H=2881:00 +CPU= 16.484
  05:26:30 STEP 2882 H=2882:00 +CPU= 17.996
  05:27:01 STEP 2883 H=2883:00 +CPU= 30.722
  05:27:22 STEP 2884 H=2884:00 +CPU= 20.304
  05:27:41 STEP 2885 H=2885:00 +CPU= 19.227
  05:28:11 STEP 2886 H=2886:00 +CPU= 28.970
  05:28:31 STEP 2887 H=2887:00 +CPU= 19.095
  05:28:50 STEP 2888 H=2888:00 +CPU= 19.561
  05:29:20 STEP 2889 H=2889:00 +CPU= 29.488
  05:29:39 STEP 2890 H=2890:00 +CPU= 18.440
  05:29:58 STEP 2891 H=2891:00 +CPU= 18.440
  05:30:32 STEP 2892 H=2892:00 +CPU= 32.391
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002892
  05:30:51 STEP 2893 H=2893:00 +CPU= 18.637
  05:31:09 STEP 2894 H=2894:00 +CPU= 18.333
  05:31:41 STEP 2895 H=2895:00 +CPU= 30.150
  05:32:01 STEP 2896 H=2896:00 +CPU= 20.309
  05:32:20 STEP 2897 H=2897:00 +CPU= 18.317
  05:32:50 STEP 2898 H=2898:00 +CPU= 28.925
  05:33:08 STEP 2899 H=2899:00 +CPU= 18.319
  05:33:26 STEP 2900 H=2900:00 +CPU= 17.932
  05:33:56 STEP 2901 H=2901:00 +CPU= 29.428
  05:34:16 STEP 2902 H=2902:00 +CPU= 19.286
  05:34:37 STEP 2903 H=2903:00 +CPU= 20.062
  05:35:14 STEP 2904 H=2904:00 +CPU= 35.503
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_120.zip
Uploading the intermediate file: upload_file_120.zip
  05:35:31 STEP 2905 H=2905:00 +CPU= 17.633
  05:35:50 STEP 2906 H=2906:00 +CPU= 18.191
  05:36:19 STEP 2907 H=2907:00 +CPU= 28.611
  05:36:40 STEP 2908 H=2908:00 +CPU= 20.651
  05:37:00 STEP 2909 H=2909:00 +CPU= 18.846
  05:37:30 STEP 2910 H=2910:00 +CPU= 29.373
  05:37:49 STEP 2911 H=2911:00 +CPU= 18.914
  05:38:08 STEP 2912 H=2912:00 +CPU= 19.106
  05:38:37 STEP 2913 H=2913:00 +CPU= 28.361
  05:38:55 STEP 2914 H=2914:00 +CPU= 16.653
  05:39:13 STEP 2915 H=2915:00 +CPU= 18.042
  05:39:46 STEP 2916 H=2916:00 +CPU= 31.567
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002916
  05:40:04 STEP 2917 H=2917:00 +CPU= 18.572
  05:40:24 STEP 2918 H=2918:00 +CPU= 18.750
  05:40:52 STEP 2919 H=2919:00 +CPU= 27.907
  05:41:13 STEP 2920 H=2920:00 +CPU= 20.119
  05:41:31 STEP 2921 H=2921:00 +CPU= 17.794
  05:42:01 STEP 2922 H=2922:00 +CPU= 29.588
  05:42:19 STEP 2923 H=2923:00 +CPU= 17.689
  05:42:40 STEP 2924 H=2924:00 +CPU= 20.337
  05:43:09 STEP 2925 H=2925:00 +CPU= 27.810
  05:43:29 STEP 2926 H=2926:00 +CPU= 19.995
  05:43:50 STEP 2927 H=2927:00 +CPU= 20.601
  05:44:26 STEP 2928 H=2928:00 +CPU= 34.822
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_121.zip
Uploading the intermediate file: upload_file_121.zip
  05:44:47 STEP 2929 H=2929:00 +CPU= 20.100
  05:45:07 STEP 2930 H=2930:00 +CPU= 20.298
  05:45:38 STEP 2931 H=2931:00 +CPU= 29.864
  05:45:59 STEP 2932 H=2932:00 +CPU= 20.650
  05:46:19 STEP 2933 H=2933:00 +CPU= 19.950
  05:46:49 STEP 2934 H=2934:00 +CPU= 28.590
  05:47:10 STEP 2935 H=2935:00 +CPU= 20.594
  05:47:30 STEP 2936 H=2936:00 +CPU= 19.601
  05:47:59 STEP 2937 H=2937:00 +CPU= 28.221
  05:48:18 STEP 2938 H=2938:00 +CPU= 18.600
  05:48:37 STEP 2939 H=2939:00 +CPU= 18.442
  05:49:09 STEP 2940 H=2940:00 +CPU= 31.411
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002940
  05:49:28 STEP 2941 H=2941:00 +CPU= 18.566
  05:49:46 STEP 2942 H=2942:00 +CPU= 17.984
  05:50:17 STEP 2943 H=2943:00 +CPU= 30.326
  05:50:39 STEP 2944 H=2944:00 +CPU= 21.169
  05:50:59 STEP 2945 H=2945:00 +CPU= 19.460
  05:51:30 STEP 2946 H=2946:00 +CPU= 30.091
  05:51:50 STEP 2947 H=2947:00 +CPU= 19.630
  05:52:10 STEP 2948 H=2948:00 +CPU= 19.933
  05:52:38 STEP 2949 H=2949:00 +CPU= 27.239
  05:52:56 STEP 2950 H=2950:00 +CPU= 17.694
  05:53:16 STEP 2951 H=2951:00 +CPU= 19.134
  05:53:49 STEP 2952 H=2952:00 +CPU= 32.347
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200942/ICMGGhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0298_2015050100_123_984_12200942_0_r1625931431_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
05:57:13 (146116): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:30:53 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 10,623,600 63,281 0.0060
09 May 2023 16:30:17 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 6,912,000 39,559 0.0057
09 May 2023 16:30:04 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 6,048,000 35,252 0.0058
09 May 2023 16:29:59 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 5,184,000 30,926 0.0060
09 May 2023 16:29:53 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 4,320,000 26,600 0.0062
09 May 2023 16:29:47 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 3,456,000 22,263 0.0064
09 May 2023 16:29:41 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 2,592,000 17,913 0.0069
09 May 2023 16:29:30 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 1,728,000 13,569 0.0079
09 May 2023 16:29:24 1532237 22287212 oifs_43r3_ps_0298_2015050100_123_984_12200942_0 864,000 9,217 0.0107


©2024 climateprediction.net