climateprediction.net home page
Task 22262478

Task 22262478

Name oifs_43r3_ps_0018_1992050100_123_961_12177662_0
Workunit 12177662
Created 21 Dec 2022, 23:19:34 UTC
Sent 21 Dec 2022, 23:33:32 UTC
Report deadline 20 Jan 2023, 23:33:32 UTC
Received 23 Dec 2022, 0:54:14 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532237
Run time 19 hours 18 min 10 sec
CPU time 15 hours 48 min 37 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.03 MB
Peak swap size 4,978.08 MB
Peak disk usage 6,867.26 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
ns@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:320954:320954] [20221222:142328:1671744208:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:24:04 STEP 2592 H=2592:00 +CPU= 32.166
  14:24:22 STEP 2593 H=2593:00 +CPU= 18.001
  14:24:38 STEP 2594 H=2594:00 +CPU= 16.079
  14:25:05 STEP 2595 H=2595:00 +CPU= 26.708
  14:25:25 STEP 2596 H=2596:00 +CPU= 19.384
  14:25:42 STEP 2597 H=2597:00 +CPU= 17.201
  14:26:12 STEP 2598 H=2598:00 +CPU= 29.496
  14:26:29 STEP 2599 H=2599:00 +CPU= 17.030
  14:26:49 STEP 2600 H=2600:00 +CPU= 19.133
  14:27:16 STEP 2601 H=2601:00 +CPU= 27.464
  14:27:34 STEP 2602 H=2602:00 +CPU= 17.810
  14:27:52 STEP 2603 H=2603:00 +CPU= 17.482
  14:28:23 STEP 2604 H=2604:00 +CPU= 30.856
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002604
  14:28:39 STEP 2605 H=2605:00 +CPU= 15.962
  14:28:57 STEP 2606 H=2606:00 +CPU= 17.436
  14:29:27 STEP 2607 H=2607:00 +CPU= 30.011
  14:29:48 STEP 2608 H=2608:00 +CPU= 19.976
  14:30:06 STEP 2609 H=2609:00 +CPU= 18.177
  14:30:35 STEP 2610 H=2610:00 +CPU= 29.166
  14:30:53 STEP 2611 H=2611:00 +CPU= 17.471
  14:31:12 STEP 2612 H=2612:00 +CPU= 18.083
  14:31:41 STEP 2613 H=2613:00 +CPU= 28.876
  14:32:01 STEP 2614 H=2614:00 +CPU= 19.416
  14:32:20 STEP 2615 H=2615:00 +CPU= 18.707
  14:32:54 STEP 2616 H=2616:00 +CPU= 33.233
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_108.zip
Uploading the intermediate file: upload_file_108.zip
  14:33:11 STEP 2617 H=2617:00 +CPU= 17.605
  14:33:28 STEP 2618 H=2618:00 +CPU= 16.156
  14:33:56 STEP 2619 H=2619:00 +CPU= 27.714
  14:34:16 STEP 2620 H=2620:00 +CPU= 20.055
  14:34:35 STEP 2621 H=2621:00 +CPU= 18.856
  14:35:04 STEP 2622 H=2622:00 +CPU= 28.066
  14:35:23 STEP 2623 H=2623:00 +CPU= 18.940
  14:35:42 STEP 2624 H=2624:00 +CPU= 18.936
  14:36:09 STEP 2625 H=2625:00 +CPU= 26.423
  14:36:28 STEP 2626 H=2626:00 +CPU= 18.992
  14:36:47 STEP 2627 H=2627:00 +CPU= 18.284
  14:37:18 STEP 2628 H=2628:00 +CPU= 31.450
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002628
  14:37:37 STEP 2629 H=2629:00 +CPU= 18.399
  14:37:56 STEP 2630 H=2630:00 +CPU= 18.687
  14:38:23 STEP 2631 H=2631:00 +CPU= 27.086
  14:38:41 STEP 2632 H=2632:00 +CPU= 17.262
  14:39:00 STEP 2633 H=2633:00 +CPU= 18.723
  14:39:30 STEP 2634 H=2634:00 +CPU= 30.012
  14:39:49 STEP 2635 H=2635:00 +CPU= 17.937
  14:40:06 STEP 2636 H=2636:00 +CPU= 17.415
  14:40:35 STEP 2637 H=2637:00 +CPU= 27.910
  14:40:54 STEP 2638 H=2638:00 +CPU= 19.290
  14:41:13 STEP 2639 H=2639:00 +CPU= 18.650
  14:41:44 STEP 2640 H=2640:00 +CPU= 29.924
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_109.zip
Uploading the intermediate file: upload_file_109.zip
  14:42:03 STEP 2641 H=2641:00 +CPU= 19.026
  14:42:22 STEP 2642 H=2642:00 +CPU= 18.199
  14:42:49 STEP 2643 H=2643:00 +CPU= 26.986
  14:43:07 STEP 2644 H=2644:00 +CPU= 18.115
  14:43:27 STEP 2645 H=2645:00 +CPU= 18.869
  14:43:55 STEP 2646 H=2646:00 +CPU= 28.731
  14:44:13 STEP 2647 H=2647:00 +CPU= 17.102
  14:44:31 STEP 2648 H=2648:00 +CPU= 17.860
  14:44:58 STEP 2649 H=2649:00 +CPU= 26.582
  14:45:17 STEP 2650 H=2650:00 +CPU= 18.757
  14:45:33 STEP 2651 H=2651:00 +CPU= 15.744
  14:46:05 STEP 2652 H=2652:00 +CPU= 31.950
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002652
  14:46:25 STEP 2653 H=2653:00 +CPU= 19.688
  14:46:47 STEP 2654 H=2654:00 +CPU= 21.259
  14:47:16 STEP 2655 H=2655:00 +CPU= 28.504
  14:47:35 STEP 2656 H=2656:00 +CPU= 18.877
  14:47:53 STEP 2657 H=2657:00 +CPU= 17.477
  14:48:22 STEP 2658 H=2658:00 +CPU= 28.867
  14:48:42 STEP 2659 H=2659:00 +CPU= 20.112
  14:49:02 STEP 2660 H=2660:00 +CPU= 19.522
  14:49:32 STEP 2661 H=2661:00 +CPU= 28.759
  14:49:52 STEP 2662 H=2662:00 +CPU= 20.161
  14:50:12 STEP 2663 H=2663:00 +CPU= 19.585
  14:50:48 STEP 2664 H=2664:00 +CPU= 35.710
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_110.zip
Uploading the intermediate file: upload_file_110.zip
  14:51:08 STEP 2665 H=2665:00 +CPU= 19.949
  14:51:29 STEP 2666 H=2666:00 +CPU= 20.512
  14:51:59 STEP 2667 H=2667:00 +CPU= 29.103
  14:52:18 STEP 2668 H=2668:00 +CPU= 19.123
  14:52:36 STEP 2669 H=2669:00 +CPU= 17.272
  14:53:04 STEP 2670 H=2670:00 +CPU= 28.168
  14:53:23 STEP 2671 H=2671:00 +CPU= 18.816
  14:53:42 STEP 2672 H=2672:00 +CPU= 17.925
  14:54:11 STEP 2673 H=2673:00 +CPU= 29.079
  14:54:31 STEP 2674 H=2674:00 +CPU= 19.450
  14:54:51 STEP 2675 H=2675:00 +CPU= 19.442
  14:55:21 STEP 2676 H=2676:00 +CPU= 30.612
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002676
  14:55:40 STEP 2677 H=2677:00 +CPU= 18.504
  14:55:57 STEP 2678 H=2678:00 +CPU= 16.849
  14:56:26 STEP 2679 H=2679:00 +CPU= 28.300
  14:56:47 STEP 2680 H=2680:00 +CPU= 20.441
  14:57:06 STEP 2681 H=2681:00 +CPU= 18.931
  14:57:35 STEP 2682 H=2682:00 +CPU= 29.223
  14:57:53 STEP 2683 H=2683:00 +CPU= 17.264
  14:58:11 STEP 2684 H=2684:00 +CPU= 17.246
  14:58:40 STEP 2685 H=2685:00 +CPU= 29.166
  14:59:00 STEP 2686 H=2686:00 +CPU= 19.095
  14:59:20 STEP 2687 H=2687:00 +CPU= 20.280
  14:59:55 STEP 2688 H=2688:00 +CPU= 34.481
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_111.zip
Uploading the intermediate file: upload_file_111.zip
  15:00:15 STEP 2689 H=2689:00 +CPU= 19.880
  15:00:35 STEP 2690 H=2690:00 +CPU= 19.210
  15:01:03 STEP 2691 H=2691:00 +CPU= 27.742
  15:01:23 STEP 2692 H=2692:00 +CPU= 19.242
  15:01:40 STEP 2693 H=2693:00 +CPU= 16.671
  15:02:08 STEP 2694 H=2694:00 +CPU= 27.724
  15:02:28 STEP 2695 H=2695:00 +CPU= 20.045
  15:02:48 STEP 2696 H=2696:00 +CPU= 19.757
  15:03:14 STEP 2697 H=2697:00 +CPU= 26.374
  15:03:34 STEP 2698 H=2698:00 +CPU= 19.240
  15:03:54 STEP 2699 H=2699:00 +CPU= 19.327
  15:04:26 STEP 2700 H=2700:00 +CPU= 31.944
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002700
  15:04:46 STEP 2701 H=2701:00 +CPU= 19.614
  15:05:08 STEP 2702 H=2702:00 +CPU= 21.209
  15:05:36 STEP 2703 H=2703:00 +CPU= 28.395
  15:05:55 STEP 2704 H=2704:00 +CPU= 18.748
  15:06:13 STEP 2705 H=2705:00 +CPU= 17.365
  15:06:42 STEP 2706 H=2706:00 +CPU= 28.897
  15:07:00 STEP 2707 H=2707:00 +CPU= 18.006
  15:07:20 STEP 2708 H=2708:00 +CPU= 19.614
  15:07:50 STEP 2709 H=2709:00 +CPU= 28.803
  15:08:09 STEP 2710 H=2710:00 +CPU= 19.008
  15:08:26 STEP 2711 H=2711:00 +CPU= 17.307
  15:09:00 STEP 2712 H=2712:00 +CPU= 32.855
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_112.zip
Uploading the intermediate file: upload_file_112.zip
  15:09:19 STEP 2713 H=2713:00 +CPU= 19.027
  15:09:37 STEP 2714 H=2714:00 +CPU= 17.928
  15:10:06 STEP 2715 H=2715:00 +CPU= 28.063
  15:10:25 STEP 2716 H=2716:00 +CPU= 18.442
  15:10:43 STEP 2717 H=2717:00 +CPU= 17.951
  15:11:13 STEP 2718 H=2718:00 +CPU= 29.773
  15:11:32 STEP 2719 H=2719:00 +CPU= 18.754
  15:11:50 STEP 2720 H=2720:00 +CPU= 18.198
  15:12:16 STEP 2721 H=2721:00 +CPU= 25.198
  15:12:37 STEP 2722 H=2722:00 +CPU= 20.242
  15:12:56 STEP 2723 H=2723:00 +CPU= 19.189
  15:13:29 STEP 2724 H=2724:00 +CPU= 32.804
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002724
  15:13:48 STEP 2725 H=2725:00 +CPU= 18.230
  15:14:06 STEP 2726 H=2726:00 +CPU= 17.546
  15:14:32 STEP 2727 H=2727:00 +CPU= 26.592
  15:14:51 STEP 2728 H=2728:00 +CPU= 18.664
  15:15:10 STEP 2729 H=2729:00 +CPU= 18.151
  15:15:38 STEP 2730 H=2730:00 +CPU= 27.492
  15:15:58 STEP 2731 H=2731:00 +CPU= 20.202
  15:16:19 STEP 2732 H=2732:00 +CPU= 20.407
  15:16:49 STEP 2733 H=2733:00 +CPU= 28.946
  15:17:07 STEP 2734 H=2734:00 +CPU= 18.678
  15:17:26 STEP 2735 H=2735:00 +CPU= 18.353
  15:17:59 STEP 2736 H=2736:00 +CPU= 32.812
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_113.zip
Uploading the intermediate file: upload_file_113.zip
  15:18:19 STEP 2737 H=2737:00 +CPU= 18.983
  15:18:38 STEP 2738 H=2738:00 +CPU= 18.948
  15:19:08 STEP 2739 H=2739:00 +CPU= 29.265
  15:19:28 STEP 2740 H=2740:00 +CPU= 19.860
  15:19:48 STEP 2741 H=2741:00 +CPU= 19.297
  15:20:17 STEP 2742 H=2742:00 +CPU= 28.721
  15:20:36 STEP 2743 H=2743:00 +CPU= 19.544
  15:20:57 STEP 2744 H=2744:00 +CPU= 20.525
  15:21:25 STEP 2745 H=2745:00 +CPU= 27.682
  15:21:44 STEP 2746 H=2746:00 +CPU= 18.524
  15:22:03 STEP 2747 H=2747:00 +CPU= 18.905
  15:22:35 STEP 2748 H=2748:00 +CPU= 31.902
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002748
  15:22:56 STEP 2749 H=2749:00 +CPU= 20.795
  15:23:17 STEP 2750 H=2750:00 +CPU= 20.546
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 961
(argv5) wuid: 12177662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_158d4079a28f7643a00f34a1d78e9328 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0018_1992050100_123_961_12177662.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0018_1992050100_123_961_12177662.zip
ic_ancil_file: ic_ancil_12177662
ifsdata_file: ifsdata_12177662
climate_data_file: clim_data_12177662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_547a86e6eb522c8f2c1fdce9eb701e93 to: /var/lib/boinc-client/slots/11/ic_ancil_12177662.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12177662.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a8b774872accbe416c852050091cf500 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12177662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12177662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12177662.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12177662.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12177662.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12177662.xml
-- Model is restarting --
Adjusting last_iter, 2751, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12177662.xml
last_cpu_time: 51729
upload_file_number: 114
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314
The child process has been launched with process id: 323963
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:323963:323963] [20221222:152441:1671747881:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:25:19 STEP 2736 H=2736:00 +CPU= 33.577
  15:25:38 STEP 2737 H=2737:00 +CPU= 19.345
  15:25:58 STEP 2738 H=2738:00 +CPU= 19.568
  15:26:27 STEP 2739 H=2739:00 +CPU= 28.405
  15:26:47 STEP 2740 H=2740:00 +CPU= 19.789
  15:27:06 STEP 2741 H=2741:00 +CPU= 18.857
  15:27:35 STEP 2742 H=2742:00 +CPU= 28.629
  15:27:52 STEP 2743 H=2743:00 +CPU= 16.932
  15:28:10 STEP 2744 H=2744:00 +CPU= 17.554
  15:28:40 STEP 2745 H=2745:00 +CPU= 29.594
  15:28:59 STEP 2746 H=2746:00 +CPU= 18.979
  15:29:17 STEP 2747 H=2747:00 +CPU= 17.663
  15:29:51 STEP 2748 H=2748:00 +CPU= 33.191
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002748
  15:30:10 STEP 2749 H=2749:00 +CPU= 19.050
  15:30:28 STEP 2750 H=2750:00 +CPU= 18.250
  15:30:57 STEP 2751 H=2751:00 +CPU= 28.042
  15:31:14 STEP 2752 H=2752:00 +CPU= 17.232
  15:31:32 STEP 2753 H=2753:00 +CPU= 17.237
  15:32:04 STEP 2754 H=2754:00 +CPU= 31.480
  15:32:22 STEP 2755 H=2755:00 +CPU= 17.557
  15:32:40 STEP 2756 H=2756:00 +CPU= 18.088
  15:33:11 STEP 2757 H=2757:00 +CPU= 30.508
  15:33:31 STEP 2758 H=2758:00 +CPU= 19.432
  15:33:51 STEP 2759 H=2759:00 +CPU= 19.506
  15:34:25 STEP 2760 H=2760:00 +CPU= 33.872
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_114.zip
Uploading the intermediate file: upload_file_114.zip
  15:34:43 STEP 2761 H=2761:00 +CPU= 17.157
  15:35:03 STEP 2762 H=2762:00 +CPU= 20.236
  15:35:35 STEP 2763 H=2763:00 +CPU= 31.409
  15:35:53 STEP 2764 H=2764:00 +CPU= 16.941
  15:36:10 STEP 2765 H=2765:00 +CPU= 17.891
  15:36:42 STEP 2766 H=2766:00 +CPU= 31.275
  15:37:00 STEP 2767 H=2767:00 +CPU= 17.739
  15:37:18 STEP 2768 H=2768:00 +CPU= 17.596
  15:37:45 STEP 2769 H=2769:00 +CPU= 27.091
  15:38:04 STEP 2770 H=2770:00 +CPU= 18.681
  15:38:23 STEP 2771 H=2771:00 +CPU= 18.026
  15:38:57 STEP 2772 H=2772:00 +CPU= 34.039
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002772
  15:39:16 STEP 2773 H=2773:00 +CPU= 18.570
  15:39:35 STEP 2774 H=2774:00 +CPU= 18.859
  15:40:06 STEP 2775 H=2775:00 +CPU= 30.005
  15:40:25 STEP 2776 H=2776:00 +CPU= 18.915
  15:40:43 STEP 2777 H=2777:00 +CPU= 17.792
  15:41:13 STEP 2778 H=2778:00 +CPU= 29.998
  15:41:32 STEP 2779 H=2779:00 +CPU= 18.201
  15:41:50 STEP 2780 H=2780:00 +CPU= 17.933
  15:42:21 STEP 2781 H=2781:00 +CPU= 30.004
  15:42:40 STEP 2782 H=2782:00 +CPU= 18.635
  15:42:58 STEP 2783 H=2783:00 +CPU= 18.664
  15:43:32 STEP 2784 H=2784:00 +CPU= 33.241
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_115.zip
Uploading the intermediate file: upload_file_115.zip
  15:43:51 STEP 2785 H=2785:00 +CPU= 18.417
  15:44:09 STEP 2786 H=2786:00 +CPU= 17.678
  15:44:38 STEP 2787 H=2787:00 +CPU= 28.744
  15:44:57 STEP 2788 H=2788:00 +CPU= 18.487
  15:45:17 STEP 2789 H=2789:00 +CPU= 19.419
  15:45:46 STEP 2790 H=2790:00 +CPU= 29.049
  15:46:02 STEP 2791 H=2791:00 +CPU= 15.419
  15:46:20 STEP 2792 H=2792:00 +CPU= 17.707
  15:46:50 STEP 2793 H=2793:00 +CPU= 30.251
  15:47:10 STEP 2794 H=2794:00 +CPU= 19.518
  15:47:29 STEP 2795 H=2795:00 +CPU= 18.541
  15:48:01 STEP 2796 H=2796:00 +CPU= 31.586
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002796
  15:48:21 STEP 2797 H=2797:00 +CPU= 19.391
  15:48:39 STEP 2798 H=2798:00 +CPU= 17.686
  15:49:10 STEP 2799 H=2799:00 +CPU= 31.096
  15:49:28 STEP 2800 H=2800:00 +CPU= 17.837
  15:49:48 STEP 2801 H=2801:00 +CPU= 18.947
  15:50:16 STEP 2802 H=2802:00 +CPU= 28.291
  15:50:35 STEP 2803 H=2803:00 +CPU= 17.939
  15:50:54 STEP 2804 H=2804:00 +CPU= 18.453
  15:51:22 STEP 2805 H=2805:00 +CPU= 28.379
  15:51:42 STEP 2806 H=2806:00 +CPU= 18.823
  15:52:00 STEP 2807 H=2807:00 +CPU= 18.486
  15:52:36 STEP 2808 H=2808:00 +CPU= 35.470
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_116.zip
Uploading the intermediate file: upload_file_116.zip
  15:52:56 STEP 2809 H=2809:00 +CPU= 19.024
  15:53:15 STEP 2810 H=2810:00 +CPU= 18.847
  15:53:45 STEP 2811 H=2811:00 +CPU= 29.207
  15:54:04 STEP 2812 H=2812:00 +CPU= 18.733
  15:54:22 STEP 2813 H=2813:00 +CPU= 17.803
  15:54:50 STEP 2814 H=2814:00 +CPU= 28.419
  15:55:07 STEP 2815 H=2815:00 +CPU= 16.889
  15:55:27 STEP 2816 H=2816:00 +CPU= 18.949
  15:55:59 STEP 2817 H=2817:00 +CPU= 31.588
  15:56:17 STEP 2818 H=2818:00 +CPU= 18.565
  15:56:36 STEP 2819 H=2819:00 +CPU= 17.943
  15:57:07 STEP 2820 H=2820:00 +CPU= 30.960
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002820
  15:57:26 STEP 2821 H=2821:00 +CPU= 19.091
  15:57:46 STEP 2822 H=2822:00 +CPU= 19.153
  15:58:16 STEP 2823 H=2823:00 +CPU= 30.174
  15:58:35 STEP 2824 H=2824:00 +CPU= 18.156
  15:58:52 STEP 2825 H=2825:00 +CPU= 17.192
  15:59:23 STEP 2826 H=2826:00 +CPU= 30.269
  15:59:42 STEP 2827 H=2827:00 +CPU= 18.217
  15:59:59 STEP 2828 H=2828:00 +CPU= 16.750
  16:00:25 STEP 2829 H=2829:00 +CPU= 26.291
  16:00:46 STEP 2830 H=2830:00 +CPU= 19.799
  16:01:04 STEP 2831 H=2831:00 +CPU= 18.502
  16:01:35 STEP 2832 H=2832:00 +CPU= 30.448
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_117.zip
Uploading the intermediate file: upload_file_117.zip
  16:01:54 STEP 2833 H=2833:00 +CPU= 18.011
  16:02:14 STEP 2834 H=2834:00 +CPU= 19.719
  16:02:46 STEP 2835 H=2835:00 +CPU= 30.982
  16:03:05 STEP 2836 H=2836:00 +CPU= 19.510
  16:03:26 STEP 2837 H=2837:00 +CPU= 20.453
  16:03:56 STEP 2838 H=2838:00 +CPU= 29.741
  16:04:14 STEP 2839 H=2839:00 +CPU= 17.983
  16:04:32 STEP 2840 H=2840:00 +CPU= 17.868
  16:05:01 STEP 2841 H=2841:00 +CPU= 28.257
  16:05:21 STEP 2842 H=2842:00 +CPU= 19.412
  16:05:39 STEP 2843 H=2843:00 +CPU= 18.116
  16:06:14 STEP 2844 H=2844:00 +CPU= 34.142
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002844
  16:06:33 STEP 2845 H=2845:00 +CPU= 18.812
  16:06:52 STEP 2846 H=2846:00 +CPU= 19.204
  16:07:22 STEP 2847 H=2847:00 +CPU= 28.807
  16:07:41 STEP 2848 H=2848:00 +CPU= 19.438
  16:08:00 STEP 2849 H=2849:00 +CPU= 18.649
  16:08:28 STEP 2850 H=2850:00 +CPU= 27.889
  16:08:49 STEP 2851 H=2851:00 +CPU= 19.762
  16:09:07 STEP 2852 H=2852:00 +CPU= 18.325
  16:09:36 STEP 2853 H=2853:00 +CPU= 28.875
  16:09:54 STEP 2854 H=2854:00 +CPU= 16.998
  16:10:13 STEP 2855 H=2855:00 +CPU= 18.693
  16:10:45 STEP 2856 H=2856:00 +CPU= 31.328
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_118.zip
Uploading the intermediate file: upload_file_118.zip
  16:11:01 STEP 2857 H=2857:00 +CPU= 16.120
  16:11:19 STEP 2858 H=2858:00 +CPU= 18.359
  16:11:50 STEP 2859 H=2859:00 +CPU= 29.774
  16:12:07 STEP 2860 H=2860:00 +CPU= 17.224
  16:12:25 STEP 2861 H=2861:00 +CPU= 17.327
  16:12:56 STEP 2862 H=2862:00 +CPU= 29.977
  16:13:13 STEP 2863 H=2863:00 +CPU= 17.759
  16:13:30 STEP 2864 H=2864:00 +CPU= 16.304
  16:13:58 STEP 2865 H=2865:00 +CPU= 27.577
  16:14:16 STEP 2866 H=2866:00 +CPU= 17.627
  16:14:34 STEP 2867 H=2867:00 +CPU= 18.602
  16:15:06 STEP 2868 H=2868:00 +CPU= 31.319
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002868
  16:15:25 STEP 2869 H=2869:00 +CPU= 18.202
  16:15:45 STEP 2870 H=2870:00 +CPU= 20.282
  16:16:16 STEP 2871 H=2871:00 +CPU= 29.815
  16:16:34 STEP 2872 H=2872:00 +CPU= 18.279
  16:16:53 STEP 2873 H=2873:00 +CPU= 18.731
  16:17:23 STEP 2874 H=2874:00 +CPU= 29.530
  16:17:41 STEP 2875 H=2875:00 +CPU= 17.466
  16:18:01 STEP 2876 H=2876:00 +CPU= 19.535
  16:18:31 STEP 2877 H=2877:00 +CPU= 29.307
  16:18:50 STEP 2878 H=2878:00 +CPU= 18.380
  16:19:10 STEP 2879 H=2879:00 +CPU= 20.706
  16:19:46 STEP 2880 H=2880:00 +CPU= 34.532
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_119.zip
Uploading the intermediate file: upload_file_119.zip
  16:20:03 STEP 2881 H=2881:00 +CPU= 17.305
  16:20:21 STEP 2882 H=2882:00 +CPU= 17.752
  16:20:52 STEP 2883 H=2883:00 +CPU= 30.390
  16:21:11 STEP 2884 H=2884:00 +CPU= 18.959
  16:21:31 STEP 2885 H=2885:00 +CPU= 19.777
  16:22:01 STEP 2886 H=2886:00 +CPU= 28.525
  16:22:19 STEP 2887 H=2887:00 +CPU= 17.779
  16:22:38 STEP 2888 H=2888:00 +CPU= 19.036
  16:23:08 STEP 2889 H=2889:00 +CPU= 29.549
  16:23:28 STEP 2890 H=2890:00 +CPU= 18.927
  16:23:48 STEP 2891 H=2891:00 +CPU= 19.938
  16:24:21 STEP 2892 H=2892:00 +CPU= 32.918
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002892
  16:24:40 STEP 2893 H=2893:00 +CPU= 18.378
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 961
(argv5) wuid: 12177662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_158d4079a28f7643a00f34a1d78e9328 to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0018_1992050100_123_961_12177662.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0018_1992050100_123_961_12177662.zip
ic_ancil_file: ic_ancil_12177662
ifsdata_file: ifsdata_12177662
climate_data_file: clim_data_12177662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_547a86e6eb522c8f2c1fdce9eb701e93 to: /var/lib/boinc-client/slots/11/ic_ancil_12177662.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12177662.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a8b774872accbe416c852050091cf500 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12177662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12177662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12177662.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12177662.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12177662.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12177662.xml
-- Model is restarting --
Adjusting last_iter, 2894, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12177662.xml
last_cpu_time: 55274
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314
The child process has been launched with process id: 327037
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:327037:327037] [20221222:162555:1671751555:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  16:26:29 STEP 2880 H=2880:00 +CPU= 31.386
  16:26:48 STEP 2881 H=2881:00 +CPU= 18.064
  16:27:07 STEP 2882 H=2882:00 +CPU= 18.991
  16:27:38 STEP 2883 H=2883:00 +CPU= 30.468
  16:27:58 STEP 2884 H=2884:00 +CPU= 19.753
  16:28:16 STEP 2885 H=2885:00 +CPU= 17.612
  16:28:44 STEP 2886 H=2886:00 +CPU= 27.806
  16:29:02 STEP 2887 H=2887:00 +CPU= 17.898
  16:29:22 STEP 2888 H=2888:00 +CPU= 19.502
  16:29:52 STEP 2889 H=2889:00 +CPU= 29.908
  16:30:10 STEP 2890 H=2890:00 +CPU= 17.793
  16:30:30 STEP 2891 H=2891:00 +CPU= 18.822
  16:30:59 STEP 2892 H=2892:00 +CPU= 29.229
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002892
  16:31:18 STEP 2893 H=2893:00 +CPU= 18.772
  16:31:37 STEP 2894 H=2894:00 +CPU= 18.908
  16:32:06 STEP 2895 H=2895:00 +CPU= 28.438
  16:32:24 STEP 2896 H=2896:00 +CPU= 17.718
  16:32:43 STEP 2897 H=2897:00 +CPU= 18.473
  16:33:12 STEP 2898 H=2898:00 +CPU= 28.842
  16:33:31 STEP 2899 H=2899:00 +CPU= 17.881
  16:33:49 STEP 2900 H=2900:00 +CPU= 18.155
  16:34:19 STEP 2901 H=2901:00 +CPU= 29.589
  16:34:37 STEP 2902 H=2902:00 +CPU= 18.235
  16:34:55 STEP 2903 H=2903:00 +CPU= 17.253
  16:35:31 STEP 2904 H=2904:00 +CPU= 35.295
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_120.zip
Uploading the intermediate file: upload_file_120.zip
  16:35:51 STEP 2905 H=2905:00 +CPU= 19.230
  16:36:09 STEP 2906 H=2906:00 +CPU= 18.118
  16:36:40 STEP 2907 H=2907:00 +CPU= 29.938
  16:36:59 STEP 2908 H=2908:00 +CPU= 19.545
  16:37:18 STEP 2909 H=2909:00 +CPU= 18.167
  16:37:49 STEP 2910 H=2910:00 +CPU= 30.665
  16:38:07 STEP 2911 H=2911:00 +CPU= 18.558
  16:38:27 STEP 2912 H=2912:00 +CPU= 19.430
  16:38:54 STEP 2913 H=2913:00 +CPU= 26.695
  16:39:15 STEP 2914 H=2914:00 +CPU= 20.599
  16:39:34 STEP 2915 H=2915:00 +CPU= 18.684
  16:40:08 STEP 2916 H=2916:00 +CPU= 33.577
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002916
  16:40:28 STEP 2917 H=2917:00 +CPU= 19.375
  16:40:47 STEP 2918 H=2918:00 +CPU= 18.987
  16:41:18 STEP 2919 H=2919:00 +CPU= 30.272
  16:41:36 STEP 2920 H=2920:00 +CPU= 17.740
  16:41:53 STEP 2921 H=2921:00 +CPU= 17.193
  16:42:21 STEP 2922 H=2922:00 +CPU= 27.168
  16:42:42 STEP 2923 H=2923:00 +CPU= 20.825
  16:43:01 STEP 2924 H=2924:00 +CPU= 19.037
  16:43:30 STEP 2925 H=2925:00 +CPU= 28.727
  16:43:50 STEP 2926 H=2926:00 +CPU= 18.785
  16:44:08 STEP 2927 H=2927:00 +CPU= 18.533
  16:44:43 STEP 2928 H=2928:00 +CPU= 33.870
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_121.zip
Uploading the intermediate file: upload_file_121.zip
  16:45:01 STEP 2929 H=2929:00 +CPU= 17.595
  16:45:22 STEP 2930 H=2930:00 +CPU= 20.518
  16:45:52 STEP 2931 H=2931:00 +CPU= 29.808
  16:46:12 STEP 2932 H=2932:00 +CPU= 20.079
  16:46:31 STEP 2933 H=2933:00 +CPU= 19.040
  16:47:00 STEP 2934 H=2934:00 +CPU= 28.036
  16:47:18 STEP 2935 H=2935:00 +CPU= 18.300
  16:47:39 STEP 2936 H=2936:00 +CPU= 20.456
  16:48:09 STEP 2937 H=2937:00 +CPU= 29.634
  16:48:28 STEP 2938 H=2938:00 +CPU= 18.561
  16:48:47 STEP 2939 H=2939:00 +CPU= 18.793
  16:49:18 STEP 2940 H=2940:00 +CPU= 30.716
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002940
  16:49:35 STEP 2941 H=2941:00 +CPU= 16.390
  16:49:55 STEP 2942 H=2942:00 +CPU= 19.727
  16:50:25 STEP 2943 H=2943:00 +CPU= 30.004
  16:50:44 STEP 2944 H=2944:00 +CPU= 18.515
  16:51:02 STEP 2945 H=2945:00 +CPU= 17.718
  16:51:31 STEP 2946 H=2946:00 +CPU= 28.477
  16:51:48 STEP 2947 H=2947:00 +CPU= 16.552
  16:52:06 STEP 2948 H=2948:00 +CPU= 18.698
  16:52:34 STEP 2949 H=2949:00 +CPU= 27.421
  16:52:53 STEP 2950 H=2950:00 +CPU= 18.667
  16:53:11 STEP 2951 H=2951:00 +CPU= 17.601
  16:53:42 STEP 2952 H=2952:00 +CPU= 30.977
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177662/ICMGGhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0018_1992050100_123_961_12177662_0_r495804314_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
16:57:06 (327030): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:49:06 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 10,623,600 56,917 0.0054
09 May 2023 15:48:22 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 6,912,000 32,879 0.0048
09 May 2023 15:48:17 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 6,048,000 29,187 0.0048
09 May 2023 15:48:12 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 5,184,000 25,289 0.0049
09 May 2023 15:48:07 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 4,320,000 21,411 0.0050
09 May 2023 15:48:02 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 3,456,000 17,286 0.0050
09 May 2023 15:47:56 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 2,592,000 13,103 0.0051
09 May 2023 15:47:51 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 1,728,000 8,930 0.0052
09 May 2023 15:47:40 1532237 22262478 oifs_43r3_ps_0018_1992050100_123_961_12177662_0 864,000 4,747 0.0055


©2024 cpdn.org