<core_client_version>7.16.16</core_client_version>
<![CDATA[
<stderr_txt>
9:13:58 STEP 2698 H=2698:00 +CPU= 45.273
09:14:42 STEP 2699 H=2699:00 +CPU= 44.005
09:15:58 STEP 2700 H=2700:00 +CPU= 75.815
Moving to projects directory: /root/slots/4/ICMGGhq0f+002700
Moving to projects directory: /root/slots/4/ICMSHhq0f+002700
Moving to projects directory: /root/slots/4/ICMUAhq0f+002700
09:16:43 STEP 2701 H=2701:00 +CPU= 44.109
09:17:29 STEP 2702 H=2702:00 +CPU= 45.456
09:18:38 STEP 2703 H=2703:00 +CPU= 68.101
09:19:24 STEP 2704 H=2704:00 +CPU= 45.448
09:20:08 STEP 2705 H=2705:00 +CPU= 43.988
09:21:19 STEP 2706 H=2706:00 +CPU= 70.098
09:22:03 STEP 2707 H=2707:00 +CPU= 43.816
09:22:49 STEP 2708 H=2708:00 +CPU= 45.340
09:23:58 STEP 2709 H=2709:00 +CPU= 68.205
09:24:44 STEP 2710 H=2710:00 +CPU= 45.458
09:25:30 STEP 2711 H=2711:00 +CPU= 45.163
09:26:51 STEP 2712 H=2712:00 +CPU= 79.813
Moving to projects directory: /root/slots/4/ICMGGhq0f+002712
Moving to projects directory: /root/slots/4/ICMSHhq0f+002712
Moving to projects directory: /root/slots/4/ICMUAhq0f+002712
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002688
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002688
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002688
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002700
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002700
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002700
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_112.zip
Uploading the intermediate file: upload_file_112.zip
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0548
(argv4) batchid: 954
(argv5) wuid: 12171192
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12171192
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4128fc5373f971631e16e0eaedf07e39 to: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
ic_ancil_file: ic_ancil_12171192
ifsdata_file: ifsdata_12171192
climate_data_file: clim_data_12171192
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c9b19b9ae9ee6662437df85134e144b to: /root/slots/4/ic_ancil_12171192.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12171192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_807dc3452e6dd6e4f7ad23cf82ae0e58 to: /root/slots/4/ifsdata/ifsdata_12171192.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12171192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12171192.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12171192.zip
Checking for progress XML file: /root/slots/4/progress_file_12171192.xml
Opened progress file ok : /root/slots/4/progress_file_12171192.xml
-- Model is restarting --
Adjusting last_iter, 2713, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12171192.xml
last_cpu_time: 151250
upload_file_number: 113
last_iter: 2711
last_upload: 9763200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634
The child process has been launched with process id: 9691
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9691:9691] [20230114:092825:1673659705:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
09:29:48 STEP 2712 H=2712:00 +CPU= 76.873
09:30:34 STEP 2713 H=2713:00 +CPU= 45.319
09:31:19 STEP 2714 H=2714:00 +CPU= 45.284
09:32:28 STEP 2715 H=2715:00 +CPU= 68.696
09:33:14 STEP 2716 H=2716:00 +CPU= 45.435
09:33:58 STEP 2717 H=2717:00 +CPU= 43.372
09:35:08 STEP 2718 H=2718:00 +CPU= 69.272
09:35:52 STEP 2719 H=2719:00 +CPU= 43.845
09:36:38 STEP 2720 H=2720:00 +CPU= 45.037
09:37:48 STEP 2721 H=2721:00 +CPU= 68.428
09:38:34 STEP 2722 H=2722:00 +CPU= 46.368
09:39:20 STEP 2723 H=2723:00 +CPU= 45.335
09:40:38 STEP 2724 H=2724:00 +CPU= 76.882
Moving to projects directory: /root/slots/4/ICMGGhq0f+002724
Moving to projects directory: /root/slots/4/ICMSHhq0f+002724
Moving to projects directory: /root/slots/4/ICMUAhq0f+002724
09:41:22 STEP 2725 H=2725:00 +CPU= 43.929
09:42:08 STEP 2726 H=2726:00 +CPU= 45.419
09:43:17 STEP 2727 H=2727:00 +CPU= 67.827
09:44:03 STEP 2728 H=2728:00 +CPU= 45.339
09:44:47 STEP 2729 H=2729:00 +CPU= 43.818
09:45:58 STEP 2730 H=2730:00 +CPU= 69.947
09:46:41 STEP 2731 H=2731:00 +CPU= 43.459
09:47:28 STEP 2732 H=2732:00 +CPU= 45.483
09:48:36 STEP 2733 H=2733:00 +CPU= 67.988
09:49:22 STEP 2734 H=2734:00 +CPU= 45.427
09:50:08 STEP 2735 H=2735:00 +CPU= 45.006
09:51:30 STEP 2736 H=2736:00 +CPU= 80.205
Moving to projects directory: /root/slots/4/ICMGGhq0f+002736
Moving to projects directory: /root/slots/4/ICMSHhq0f+002736
Moving to projects directory: /root/slots/4/ICMUAhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002712
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002712
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002712
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002724
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002724
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002724
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_113.zip
Uploading the intermediate file: upload_file_113.zip
09:52:14 STEP 2737 H=2737:00 +CPU= 43.880
09:53:01 STEP 2738 H=2738:00 +CPU= 45.803
09:54:10 STEP 2739 H=2739:00 +CPU= 68.003
09:54:56 STEP 2740 H=2740:00 +CPU= 46.020
09:55:40 STEP 2741 H=2741:00 +CPU= 43.907
09:56:51 STEP 2742 H=2742:00 +CPU= 69.917
09:57:36 STEP 2743 H=2743:00 +CPU= 44.306
09:58:21 STEP 2744 H=2744:00 +CPU= 45.427
09:59:31 STEP 2745 H=2745:00 +CPU= 68.442
10:00:17 STEP 2746 H=2746:00 +CPU= 45.598
10:01:02 STEP 2747 H=2747:00 +CPU= 43.930
10:02:20 STEP 2748 H=2748:00 +CPU= 77.198
Moving to projects directory: /root/slots/4/ICMGGhq0f+002748
Moving to projects directory: /root/slots/4/ICMSHhq0f+002748
Moving to projects directory: /root/slots/4/ICMUAhq0f+002748
10:03:06 STEP 2749 H=2749:00 +CPU= 45.568
10:03:53 STEP 2750 H=2750:00 +CPU= 46.226
10:05:03 STEP 2751 H=2751:00 +CPU= 69.330
10:05:50 STEP 2752 H=2752:00 +CPU= 47.029
10:06:36 STEP 2753 H=2753:00 +CPU= 45.761
10:07:51 STEP 2754 H=2754:00 +CPU= 73.757
10:08:35 STEP 2755 H=2755:00 +CPU= 44.284
10:09:23 STEP 2756 H=2756:00 +CPU= 46.447
10:10:33 STEP 2757 H=2757:00 +CPU= 69.889
10:11:20 STEP 2758 H=2758:00 +CPU= 46.270
10:12:08 STEP 2759 H=2759:00 +CPU= 47.700
10:13:30 STEP 2760 H=2760:00 +CPU= 81.401
Moving to projects directory: /root/slots/4/ICMGGhq0f+002760
Moving to projects directory: /root/slots/4/ICMSHhq0f+002760
Moving to projects directory: /root/slots/4/ICMUAhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002736
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002748
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002748
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002748
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_114.zip
Uploading the intermediate file: upload_file_114.zip
10:14:15 STEP 2761 H=2761:00 +CPU= 44.392
10:15:02 STEP 2762 H=2762:00 +CPU= 46.306
10:16:12 STEP 2763 H=2763:00 +CPU= 69.210
10:16:58 STEP 2764 H=2764:00 +CPU= 45.996
10:17:46 STEP 2765 H=2765:00 +CPU= 47.133
10:18:59 STEP 2766 H=2766:00 +CPU= 73.210
10:19:46 STEP 2767 H=2767:00 +CPU= 45.994
10:20:35 STEP 2768 H=2768:00 +CPU= 49.208
10:21:48 STEP 2769 H=2769:00 +CPU= 72.287
10:22:34 STEP 2770 H=2770:00 +CPU= 45.587
10:23:19 STEP 2771 H=2771:00 +CPU= 44.822
10:24:39 STEP 2772 H=2772:00 +CPU= 79.107
Moving to projects directory: /root/slots/4/ICMGGhq0f+002772
Moving to projects directory: /root/slots/4/ICMSHhq0f+002772
Moving to projects directory: /root/slots/4/ICMUAhq0f+002772
10:25:26 STEP 2773 H=2773:00 +CPU= 46.735
10:26:14 STEP 2774 H=2774:00 +CPU= 47.547
10:27:24 STEP 2775 H=2775:00 +CPU= 69.549
10:28:12 STEP 2776 H=2776:00 +CPU= 46.863
10:28:58 STEP 2777 H=2777:00 +CPU= 45.592
10:30:10 STEP 2778 H=2778:00 +CPU= 71.287
10:30:57 STEP 2779 H=2779:00 +CPU= 46.927
10:31:43 STEP 2780 H=2780:00 +CPU= 46.161
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0548
(argv4) batchid: 954
(argv5) wuid: 12171192
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12171192
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4128fc5373f971631e16e0eaedf07e39 to: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
ic_ancil_file: ic_ancil_12171192
ifsdata_file: ifsdata_12171192
climate_data_file: clim_data_12171192
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c9b19b9ae9ee6662437df85134e144b to: /root/slots/4/ic_ancil_12171192.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12171192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_807dc3452e6dd6e4f7ad23cf82ae0e58 to: /root/slots/4/ifsdata/ifsdata_12171192.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12171192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12171192.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12171192.zip
Checking for progress XML file: /root/slots/4/progress_file_12171192.xml
Opened progress file ok : /root/slots/4/progress_file_12171192.xml
-- Model is restarting --
Adjusting last_iter, 2781, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12171192.xml
last_cpu_time: 155025
upload_file_number: 115
last_iter: 2759
last_upload: 9936000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634
The child process has been launched with process id: 9740
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9740:9740] [20230114:103818:1673663898:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
10:39:43 STEP 2760 H=2760:00 +CPU= 79.027
10:40:29 STEP 2761 H=2761:00 +CPU= 44.877
10:41:16 STEP 2762 H=2762:00 +CPU= 47.134
10:42:27 STEP 2763 H=2763:00 +CPU= 69.618
10:43:13 STEP 2764 H=2764:00 +CPU= 45.911
10:43:58 STEP 2765 H=2765:00 +CPU= 44.600
10:45:12 STEP 2766 H=2766:00 +CPU= 72.849
10:45:57 STEP 2767 H=2767:00 +CPU= 45.203
10:46:44 STEP 2768 H=2768:00 +CPU= 46.806
10:47:55 STEP 2769 H=2769:00 +CPU= 69.731
10:48:44 STEP 2770 H=2770:00 +CPU= 47.974
10:49:30 STEP 2771 H=2771:00 +CPU= 45.483
10:50:50 STEP 2772 H=2772:00 +CPU= 79.215
Moving to projects directory: /root/slots/4/ICMGGhq0f+002772
Moving to projects directory: /root/slots/4/ICMSHhq0f+002772
Moving to projects directory: /root/slots/4/ICMUAhq0f+002772
10:51:37 STEP 2773 H=2773:00 +CPU= 46.126
10:52:25 STEP 2774 H=2774:00 +CPU= 47.448
10:53:35 STEP 2775 H=2775:00 +CPU= 69.020
10:54:22 STEP 2776 H=2776:00 +CPU= 47.086
10:55:07 STEP 2777 H=2777:00 +CPU= 44.845
10:56:19 STEP 2778 H=2778:00 +CPU= 71.133
10:57:04 STEP 2779 H=2779:00 +CPU= 44.466
10:57:51 STEP 2780 H=2780:00 +CPU= 46.966
10:59:01 STEP 2781 H=2781:00 +CPU= 68.773
10:59:48 STEP 2782 H=2782:00 +CPU= 46.078
11:00:35 STEP 2783 H=2783:00 +CPU= 46.429
11:01:59 STEP 2784 H=2784:00 +CPU= 82.491
Moving to projects directory: /root/slots/4/ICMGGhq0f+002784
Moving to projects directory: /root/slots/4/ICMSHhq0f+002784
Moving to projects directory: /root/slots/4/ICMUAhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002760
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002772
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002772
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002772
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_115.zip
Uploading the intermediate file: upload_file_115.zip
11:02:44 STEP 2785 H=2785:00 +CPU= 43.746
11:03:30 STEP 2786 H=2786:00 +CPU= 45.363
11:04:38 STEP 2787 H=2787:00 +CPU= 67.888
11:05:24 STEP 2788 H=2788:00 +CPU= 45.597
11:06:09 STEP 2789 H=2789:00 +CPU= 43.615
11:07:20 STEP 2790 H=2790:00 +CPU= 70.453
11:08:04 STEP 2791 H=2791:00 +CPU= 43.897
11:08:50 STEP 2792 H=2792:00 +CPU= 45.442
11:09:59 STEP 2793 H=2793:00 +CPU= 68.037
11:10:45 STEP 2794 H=2794:00 +CPU= 45.259
11:11:29 STEP 2795 H=2795:00 +CPU= 43.484
11:12:46 STEP 2796 H=2796:00 +CPU= 75.870
Moving to projects directory: /root/slots/4/ICMGGhq0f+002796
Moving to projects directory: /root/slots/4/ICMSHhq0f+002796
Moving to projects directory: /root/slots/4/ICMUAhq0f+002796
11:13:30 STEP 2797 H=2797:00 +CPU= 43.952
11:14:16 STEP 2798 H=2798:00 +CPU= 45.351
11:15:24 STEP 2799 H=2799:00 +CPU= 67.725
11:16:10 STEP 2800 H=2800:00 +CPU= 46.118
11:16:55 STEP 2801 H=2801:00 +CPU= 44.546
11:18:06 STEP 2802 H=2802:00 +CPU= 70.310
11:18:51 STEP 2803 H=2803:00 +CPU= 44.153
11:19:37 STEP 2804 H=2804:00 +CPU= 45.398
11:20:46 STEP 2805 H=2805:00 +CPU= 68.888
11:21:34 STEP 2806 H=2806:00 +CPU= 46.743
11:22:20 STEP 2807 H=2807:00 +CPU= 45.524
11:23:43 STEP 2808 H=2808:00 +CPU= 81.791
Moving to projects directory: /root/slots/4/ICMGGhq0f+002808
Moving to projects directory: /root/slots/4/ICMSHhq0f+002808
Moving to projects directory: /root/slots/4/ICMUAhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002784
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002796
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002796
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002796
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_116.zip
Uploading the intermediate file: upload_file_116.zip
11:24:29 STEP 2809 H=2809:00 +CPU= 44.685
11:25:16 STEP 2810 H=2810:00 +CPU= 46.804
11:26:27 STEP 2811 H=2811:00 +CPU= 69.812
11:27:14 STEP 2812 H=2812:00 +CPU= 47.500
11:28:01 STEP 2813 H=2813:00 +CPU= 46.703
11:29:13 STEP 2814 H=2814:00 +CPU= 70.887
11:29:57 STEP 2815 H=2815:00 +CPU= 44.113
11:30:43 STEP 2816 H=2816:00 +CPU= 45.917
11:31:52 STEP 2817 H=2817:00 +CPU= 68.258
11:32:40 STEP 2818 H=2818:00 +CPU= 46.926
11:33:25 STEP 2819 H=2819:00 +CPU= 44.814
11:34:43 STEP 2820 H=2820:00 +CPU= 77.728
Moving to projects directory: /root/slots/4/ICMGGhq0f+002820
Moving to projects directory: /root/slots/4/ICMSHhq0f+002820
Moving to projects directory: /root/slots/4/ICMUAhq0f+002820
11:35:28 STEP 2821 H=2821:00 +CPU= 44.357
11:36:15 STEP 2822 H=2822:00 +CPU= 45.923
11:37:27 STEP 2823 H=2823:00 +CPU= 70.768
11:38:13 STEP 2824 H=2824:00 +CPU= 45.819
11:38:58 STEP 2825 H=2825:00 +CPU= 44.797
11:40:10 STEP 2826 H=2826:00 +CPU= 71.166
11:40:55 STEP 2827 H=2827:00 +CPU= 44.864
11:41:43 STEP 2828 H=2828:00 +CPU= 46.243
11:42:54 STEP 2829 H=2829:00 +CPU= 70.449
11:43:41 STEP 2830 H=2830:00 +CPU= 46.216
11:44:28 STEP 2831 H=2831:00 +CPU= 46.377
11:45:51 STEP 2832 H=2832:00 +CPU= 82.075
Moving to projects directory: /root/slots/4/ICMGGhq0f+002832
Moving to projects directory: /root/slots/4/ICMSHhq0f+002832
Moving to projects directory: /root/slots/4/ICMUAhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002808
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002820
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002820
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002820
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_117.zip
Uploading the intermediate file: upload_file_117.zip
11:46:37 STEP 2833 H=2833:00 +CPU= 44.920
11:47:24 STEP 2834 H=2834:00 +CPU= 46.273
11:48:34 STEP 2835 H=2835:00 +CPU= 69.616
11:49:21 STEP 2836 H=2836:00 +CPU= 46.338
11:50:06 STEP 2837 H=2837:00 +CPU= 44.439
11:51:19 STEP 2838 H=2838:00 +CPU= 71.753
11:52:04 STEP 2839 H=2839:00 +CPU= 44.322
11:52:50 STEP 2840 H=2840:00 +CPU= 46.458
11:54:02 STEP 2841 H=2841:00 +CPU= 69.989
11:54:49 STEP 2842 H=2842:00 +CPU= 46.589
11:55:35 STEP 2843 H=2843:00 +CPU= 44.977
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0548
(argv4) batchid: 954
(argv5) wuid: 12171192
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12171192
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4128fc5373f971631e16e0eaedf07e39 to: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
ic_ancil_file: ic_ancil_12171192
ifsdata_file: ifsdata_12171192
climate_data_file: clim_data_12171192
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c9b19b9ae9ee6662437df85134e144b to: /root/slots/4/ic_ancil_12171192.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12171192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_807dc3452e6dd6e4f7ad23cf82ae0e58 to: /root/slots/4/ifsdata/ifsdata_12171192.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12171192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12171192.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12171192.zip
Checking for progress XML file: /root/slots/4/progress_file_12171192.xml
Opened progress file ok : /root/slots/4/progress_file_12171192.xml
-- Model is restarting --
Adjusting last_iter, 2844, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12171192.xml
last_cpu_time: 159650
upload_file_number: 118
last_iter: 2831
last_upload: 10195200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634
The child process has been launched with process id: 9865
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9865:9865] [20230114:120103:1673668863:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
12:02:32 STEP 2832 H=2832:00 +CPU= 81.630
12:03:20 STEP 2833 H=2833:00 +CPU= 47.022
12:04:07 STEP 2834 H=2834:00 +CPU= 47.210
12:05:19 STEP 2835 H=2835:00 +CPU= 69.989
12:06:06 STEP 2836 H=2836:00 +CPU= 46.862
12:06:51 STEP 2837 H=2837:00 +CPU= 44.881
12:08:03 STEP 2838 H=2838:00 +CPU= 71.869
12:08:51 STEP 2839 H=2839:00 +CPU= 46.592
12:09:40 STEP 2840 H=2840:00 +CPU= 48.911
12:10:54 STEP 2841 H=2841:00 +CPU= 72.361
12:11:44 STEP 2842 H=2842:00 +CPU= 49.695
12:12:30 STEP 2843 H=2843:00 +CPU= 45.750
12:13:52 STEP 2844 H=2844:00 +CPU= 80.563
Moving to projects directory: /root/slots/4/ICMGGhq0f+002844
Moving to projects directory: /root/slots/4/ICMSHhq0f+002844
Moving to projects directory: /root/slots/4/ICMUAhq0f+002844
12:14:38 STEP 2845 H=2845:00 +CPU= 45.559
12:15:27 STEP 2846 H=2846:00 +CPU= 48.669
12:16:40 STEP 2847 H=2847:00 +CPU= 72.094
12:17:28 STEP 2848 H=2848:00 +CPU= 47.858
12:18:15 STEP 2849 H=2849:00 +CPU= 46.358
12:19:29 STEP 2850 H=2850:00 +CPU= 72.897
12:20:15 STEP 2851 H=2851:00 +CPU= 45.835
12:21:02 STEP 2852 H=2852:00 +CPU= 46.301
12:22:11 STEP 2853 H=2853:00 +CPU= 68.429
12:22:56 STEP 2854 H=2854:00 +CPU= 45.210
12:23:42 STEP 2855 H=2855:00 +CPU= 45.108
12:25:05 STEP 2856 H=2856:00 +CPU= 80.458
Moving to projects directory: /root/slots/4/ICMGGhq0f+002856
Moving to projects directory: /root/slots/4/ICMSHhq0f+002856
Moving to projects directory: /root/slots/4/ICMUAhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002832
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002844
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002844
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002844
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_118.zip
Uploading the intermediate file: upload_file_118.zip
12:25:50 STEP 2857 H=2857:00 +CPU= 45.316
12:26:38 STEP 2858 H=2858:00 +CPU= 47.140
12:27:49 STEP 2859 H=2859:00 +CPU= 70.301
12:28:37 STEP 2860 H=2860:00 +CPU= 46.812
12:29:23 STEP 2861 H=2861:00 +CPU= 45.246
12:30:36 STEP 2862 H=2862:00 +CPU= 72.652
12:31:22 STEP 2863 H=2863:00 +CPU= 45.784
12:32:12 STEP 2864 H=2864:00 +CPU= 48.274
12:33:28 STEP 2865 H=2865:00 +CPU= 74.662
12:34:18 STEP 2866 H=2866:00 +CPU= 49.154
12:35:06 STEP 2867 H=2867:00 +CPU= 47.093
12:36:28 STEP 2868 H=2868:00 +CPU= 81.433
Moving to projects directory: /root/slots/4/ICMGGhq0f+002868
Moving to projects directory: /root/slots/4/ICMSHhq0f+002868
Moving to projects directory: /root/slots/4/ICMUAhq0f+002868
12:37:16 STEP 2869 H=2869:00 +CPU= 47.235
12:38:05 STEP 2870 H=2870:00 +CPU= 48.742
12:39:16 STEP 2871 H=2871:00 +CPU= 70.201
12:40:04 STEP 2872 H=2872:00 +CPU= 47.055
12:40:50 STEP 2873 H=2873:00 +CPU= 46.428
12:42:05 STEP 2874 H=2874:00 +CPU= 73.829
12:42:51 STEP 2875 H=2875:00 +CPU= 45.675
12:43:38 STEP 2876 H=2876:00 +CPU= 46.574
12:44:48 STEP 2877 H=2877:00 +CPU= 69.565
12:45:36 STEP 2878 H=2878:00 +CPU= 46.693
12:46:22 STEP 2879 H=2879:00 +CPU= 45.871
12:47:45 STEP 2880 H=2880:00 +CPU= 81.383
Moving to projects directory: /root/slots/4/ICMGGhq0f+002880
Moving to projects directory: /root/slots/4/ICMSHhq0f+002880
Moving to projects directory: /root/slots/4/ICMUAhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002856
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002868
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002868
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002868
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_119.zip
Uploading the intermediate file: upload_file_119.zip
12:48:30 STEP 2881 H=2881:00 +CPU= 44.767
12:49:19 STEP 2882 H=2882:00 +CPU= 48.178
12:50:30 STEP 2883 H=2883:00 +CPU= 69.715
12:51:17 STEP 2884 H=2884:00 +CPU= 47.519
12:52:06 STEP 2885 H=2885:00 +CPU= 48.071
12:53:18 STEP 2886 H=2886:00 +CPU= 71.204
12:54:04 STEP 2887 H=2887:00 +CPU= 45.757
12:54:50 STEP 2888 H=2888:00 +CPU= 45.737
12:56:01 STEP 2889 H=2889:00 +CPU= 69.823
12:56:48 STEP 2890 H=2890:00 +CPU= 46.542
12:57:33 STEP 2891 H=2891:00 +CPU= 44.345
12:58:52 STEP 2892 H=2892:00 +CPU= 78.158
Moving to projects directory: /root/slots/4/ICMGGhq0f+002892
Moving to projects directory: /root/slots/4/ICMSHhq0f+002892
Moving to projects directory: /root/slots/4/ICMUAhq0f+002892
12:59:38 STEP 2893 H=2893:00 +CPU= 45.481
13:00:25 STEP 2894 H=2894:00 +CPU= 46.552
13:01:36 STEP 2895 H=2895:00 +CPU= 69.705
13:02:23 STEP 2896 H=2896:00 +CPU= 47.133
13:03:09 STEP 2897 H=2897:00 +CPU= 45.651
13:04:21 STEP 2898 H=2898:00 +CPU= 71.183
13:05:07 STEP 2899 H=2899:00 +CPU= 45.433
13:05:55 STEP 2900 H=2900:00 +CPU= 46.933
13:07:06 STEP 2901 H=2901:00 +CPU= 70.445
13:07:53 STEP 2902 H=2902:00 +CPU= 47.040
13:08:41 STEP 2903 H=2903:00 +CPU= 47.555
13:10:04 STEP 2904 H=2904:00 +CPU= 81.352
Moving to projects directory: /root/slots/4/ICMGGhq0f+002904
Moving to projects directory: /root/slots/4/ICMSHhq0f+002904
Moving to projects directory: /root/slots/4/ICMUAhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002880
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002892
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002892
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002892
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_120.zip
Uploading the intermediate file: upload_file_120.zip
13:10:50 STEP 2905 H=2905:00 +CPU= 45.011
13:11:36 STEP 2906 H=2906:00 +CPU= 45.919
13:12:46 STEP 2907 H=2907:00 +CPU= 69.155
13:13:33 STEP 2908 H=2908:00 +CPU= 46.301
13:14:18 STEP 2909 H=2909:00 +CPU= 44.212
13:15:31 STEP 2910 H=2910:00 +CPU= 71.680
13:16:16 STEP 2911 H=2911:00 +CPU= 45.029
13:17:04 STEP 2912 H=2912:00 +CPU= 46.953
13:18:14 STEP 2913 H=2913:00 +CPU= 69.086
13:19:02 STEP 2914 H=2914:00 +CPU= 47.700
13:19:49 STEP 2915 H=2915:00 +CPU= 45.631
13:21:08 STEP 2916 H=2916:00 +CPU= 78.341
Moving to projects directory: /root/slots/4/ICMGGhq0f+002916
Moving to projects directory: /root/slots/4/ICMSHhq0f+002916
Moving to projects directory: /root/slots/4/ICMUAhq0f+002916
13:21:53 STEP 2917 H=2917:00 +CPU= 44.121
13:22:40 STEP 2918 H=2918:00 +CPU= 46.899
13:23:51 STEP 2919 H=2919:00 +CPU= 69.943
13:24:37 STEP 2920 H=2920:00 +CPU= 45.766
13:25:23 STEP 2921 H=2921:00 +CPU= 45.109
13:26:35 STEP 2922 H=2922:00 +CPU= 71.137
13:27:20 STEP 2923 H=2923:00 +CPU= 43.820
13:28:06 STEP 2924 H=2924:00 +CPU= 45.876
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1985050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0548
(argv4) batchid: 954
(argv5) wuid: 12171192
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/4
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12171192
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4128fc5373f971631e16e0eaedf07e39 to: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
Unzipping the namelist zip file: /root/slots/4/oifs_43r3_ps_0548_1985050100_123_954_12171192.zip
ic_ancil_file: ic_ancil_12171192
ifsdata_file: ifsdata_12171192
climate_data_file: clim_data_12171192
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8c9b19b9ae9ee6662437df85134e144b to: /root/slots/4/ic_ancil_12171192.zip
Unzipping the IC ancils zip file: /root/slots/4/ic_ancil_12171192.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_807dc3452e6dd6e4f7ad23cf82ae0e58 to: /root/slots/4/ifsdata/ifsdata_12171192.zip
Unzipping the ifsdata_zip file: /root/slots/4/ifsdata/ifsdata_12171192.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/4/159l_2/clim_data_12171192.zip
Unzipping the climate data zip file: /root/slots/4/159l_2/clim_data_12171192.zip
Checking for progress XML file: /root/slots/4/progress_file_12171192.xml
Opened progress file ok : /root/slots/4/progress_file_12171192.xml
-- Model is restarting --
Adjusting last_iter, 2925, to previous model restart step.
Creating progress file: /root/slots/4/progress_file_12171192.xml
last_cpu_time: 164818
upload_file_number: 121
last_iter: 2903
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634
The child process has been launched with process id: 9916
Executing the command: /root/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PTSH-BOINC1:1:1:9916:9916] [20230114:134428:1673675068:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
13:45:58 STEP 2904 H=2904:00 +CPU= 83.801
13:46:45 STEP 2905 H=2905:00 +CPU= 46.594
13:47:33 STEP 2906 H=2906:00 +CPU= 47.216
13:48:44 STEP 2907 H=2907:00 +CPU= 70.540
13:49:32 STEP 2908 H=2908:00 +CPU= 47.247
13:50:19 STEP 2909 H=2909:00 +CPU= 45.842
13:51:33 STEP 2910 H=2910:00 +CPU= 73.735
13:52:35 STEP 2911 H=2911:00 +CPU= 48.964
13:53:26 STEP 2912 H=2912:00 +CPU= 48.810
13:54:39 STEP 2913 H=2913:00 +CPU= 72.146
13:55:28 STEP 2914 H=2914:00 +CPU= 48.702
13:56:15 STEP 2915 H=2915:00 +CPU= 46.368
13:57:37 STEP 2916 H=2916:00 +CPU= 80.374
Moving to projects directory: /root/slots/4/ICMGGhq0f+002916
Moving to projects directory: /root/slots/4/ICMSHhq0f+002916
Moving to projects directory: /root/slots/4/ICMUAhq0f+002916
13:58:25 STEP 2917 H=2917:00 +CPU= 47.080
13:59:18 STEP 2918 H=2918:00 +CPU= 49.360
14:00:29 STEP 2919 H=2919:00 +CPU= 70.794
14:01:17 STEP 2920 H=2920:00 +CPU= 47.721
14:02:04 STEP 2921 H=2921:00 +CPU= 45.985
14:03:18 STEP 2922 H=2922:00 +CPU= 73.134
14:04:07 STEP 2923 H=2923:00 +CPU= 48.430
14:04:55 STEP 2924 H=2924:00 +CPU= 47.891
14:06:08 STEP 2925 H=2925:00 +CPU= 71.449
14:06:57 STEP 2926 H=2926:00 +CPU= 48.780
14:07:46 STEP 2927 H=2927:00 +CPU= 47.744
14:09:13 STEP 2928 H=2928:00 +CPU= 83.936
Moving to projects directory: /root/slots/4/ICMGGhq0f+002928
Moving to projects directory: /root/slots/4/ICMSHhq0f+002928
Moving to projects directory: /root/slots/4/ICMUAhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002904
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002916
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002916
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002916
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_121.zip
Uploading the intermediate file: upload_file_121.zip
14:10:02 STEP 2929 H=2929:00 +CPU= 47.524
14:10:49 STEP 2930 H=2930:00 +CPU= 47.107
14:12:00 STEP 2931 H=2931:00 +CPU= 69.956
14:12:48 STEP 2932 H=2932:00 +CPU= 47.045
14:13:33 STEP 2933 H=2933:00 +CPU= 44.797
14:14:45 STEP 2934 H=2934:00 +CPU= 71.512
14:15:31 STEP 2935 H=2935:00 +CPU= 44.882
14:16:20 STEP 2936 H=2936:00 +CPU= 48.437
14:17:31 STEP 2937 H=2937:00 +CPU= 70.565
14:18:18 STEP 2938 H=2938:00 +CPU= 45.793
14:19:03 STEP 2939 H=2939:00 +CPU= 44.587
14:20:21 STEP 2940 H=2940:00 +CPU= 77.243
Moving to projects directory: /root/slots/4/ICMGGhq0f+002940
Moving to projects directory: /root/slots/4/ICMSHhq0f+002940
Moving to projects directory: /root/slots/4/ICMUAhq0f+002940
14:21:05 STEP 2941 H=2941:00 +CPU= 44.081
14:21:52 STEP 2942 H=2942:00 +CPU= 46.505
14:23:04 STEP 2943 H=2943:00 +CPU= 71.653
14:23:51 STEP 2944 H=2944:00 +CPU= 46.218
14:24:37 STEP 2945 H=2945:00 +CPU= 45.381
14:25:50 STEP 2946 H=2946:00 +CPU= 72.302
14:26:36 STEP 2947 H=2947:00 +CPU= 45.363
14:27:24 STEP 2948 H=2948:00 +CPU= 47.517
14:28:33 STEP 2949 H=2949:00 +CPU= 68.860
14:29:19 STEP 2950 H=2950:00 +CPU= 45.491
14:30:05 STEP 2951 H=2951:00 +CPU= 45.161
14:31:25 STEP 2952 H=2952:00 +CPU= 77.723
The child process terminated with status: 0
Moving to projects directory: /root/slots/4/ICMGGhq0f+002952
Moving to projects directory: /root/slots/4/ICMSHhq0f+002952
Moving to projects directory: /root/slots/4/ICMUAhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002940
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002940
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002940
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002928
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMSHhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMGGhq0f+002952
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12171192/ICMUAhq0f+002928
Zipping up the final file: /root/projects/climateprediction.net/oifs_43r3_ps_0548_1985050100_123_954_12171192_1_r1326278634_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
14:34:48 (9913): called boinc_finish(0)
</stderr_txt>
]]>
