<core_client_version>7.21.0</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
10:33:20 STEP 962 H= 962:00 +CPU= 17.061
10:33:49 STEP 963 H= 963:00 +CPU= 27.795
10:34:07 STEP 964 H= 964:00 +CPU= 17.547
10:34:23 STEP 965 H= 965:00 +CPU= 16.529
10:34:51 STEP 966 H= 966:00 +CPU= 28.042
10:35:09 STEP 967 H= 967:00 +CPU= 17.019
10:35:26 STEP 968 H= 968:00 +CPU= 17.307
10:35:53 STEP 969 H= 969:00 +CPU= 26.690
10:36:10 STEP 970 H= 970:00 +CPU= 16.951
10:36:27 STEP 971 H= 971:00 +CPU= 16.241
10:36:56 STEP 972 H= 972:00 +CPU= 29.556
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+000972
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+000972
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+000972
10:37:13 STEP 973 H= 973:00 +CPU= 16.420
10:37:30 STEP 974 H= 974:00 +CPU= 16.982
10:37:57 STEP 975 H= 975:00 +CPU= 27.127
10:38:15 STEP 976 H= 976:00 +CPU= 17.528
10:38:33 STEP 977 H= 977:00 +CPU= 16.987
10:39:01 STEP 978 H= 978:00 +CPU= 28.118
10:39:18 STEP 979 H= 979:00 +CPU= 16.931
10:39:36 STEP 980 H= 980:00 +CPU= 17.415
10:40:03 STEP 981 H= 981:00 +CPU= 27.443
10:40:20 STEP 982 H= 982:00 +CPU= 16.665
10:40:37 STEP 983 H= 983:00 +CPU= 16.580
10:41:07 STEP 984 H= 984:00 +CPU= 30.126
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+000984
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+000984
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+000984
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+000960
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+000960
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+000960
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+000972
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+000972
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+000972
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_40.zip
Uploading the intermediate file: upload_file_40.zip
10:41:23 STEP 985 H= 985:00 +CPU= 16.109
Uploading trickle at timestep: 3542400
10:41:40 STEP 986 H= 986:00 +CPU= 16.615
10:42:06 STEP 987 H= 987:00 +CPU= 26.137
10:42:23 STEP 988 H= 988:00 +CPU= 16.699
10:42:39 STEP 989 H= 989:00 +CPU= 16.079
10:43:06 STEP 990 H= 990:00 +CPU= 26.795
10:43:22 STEP 991 H= 991:00 +CPU= 16.121
10:43:39 STEP 992 H= 992:00 +CPU= 16.637
10:44:05 STEP 993 H= 993:00 +CPU= 26.119
10:44:21 STEP 994 H= 994:00 +CPU= 16.640
10:44:37 STEP 995 H= 995:00 +CPU= 16.082
10:45:07 STEP 996 H= 996:00 +CPU= 29.043
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+000996
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+000996
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+000996
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:57:39 STEP 997 H= 997:00 +CPU= 18.917
10:58:03 STEP 998 H= 998:00 +CPU= 23.021
10:58:40 STEP 999 H= 999:00 +CPU= 35.823
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:14:36 STEP 1000 H=1000:00 +CPU= 20.898
11:14:56 STEP 1001 H=1001:00 +CPU= 19.294
11:15:28 STEP 1002 H=1002:00 +CPU= 31.090
11:15:47 STEP 1003 H=1003:00 +CPU= 19.097
11:16:07 STEP 1004 H=1004:00 +CPU= 19.484
11:16:39 STEP 1005 H=1005:00 +CPU= 31.117
11:17:00 STEP 1006 H=1006:00 +CPU= 20.051
11:17:20 STEP 1007 H=1007:00 +CPU= 19.290
11:17:57 STEP 1008 H=1008:00 +CPU= 36.354
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001008
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001008
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001008
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+000984
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+000984
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+000984
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+000996
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+000996
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+000996
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_41.zip
Uploading the intermediate file: upload_file_41.zip
11:18:15 STEP 1009 H=1009:00 +CPU= 17.261
11:18:32 STEP 1010 H=1010:00 +CPU= 17.503
11:19:01 STEP 1011 H=1011:00 +CPU= 29.011
11:19:20 STEP 1012 H=1012:00 +CPU= 18.429
11:19:37 STEP 1013 H=1013:00 +CPU= 17.286
11:20:07 STEP 1014 H=1014:00 +CPU= 29.493
11:20:25 STEP 1015 H=1015:00 +CPU= 17.985
11:20:43 STEP 1016 H=1016:00 +CPU= 18.095
11:21:11 STEP 1017 H=1017:00 +CPU= 28.037
11:21:29 STEP 1018 H=1018:00 +CPU= 17.833
11:21:47 STEP 1019 H=1019:00 +CPU= 17.670
Quit request received from BOINC client, ending the child process
11:22:05 (22415): BOINC client no longer exists - exiting
11:22:05 (22415): timer handler: client dead, exiting
11:22:10 (22415): BOINC client no longer exists - exiting
11:22:10 (22415): timer handler: client dead, exiting
11:22:15 (22415): BOINC client no longer exists - exiting
11:22:15 (22415): timer handler: client dead, exiting
11:22:20 (22415): BOINC client no longer exists - exiting
11:22:20 (22415): timer handler: client dead, exiting
11:22:25 (22415): BOINC client no longer exists - exiting
11:22:25 (22415): timer handler: client dead, exiting
11:22:30 (22415): BOINC client no longer exists - exiting
11:22:30 (22415): timer handler: client dead, exiting
11:22:35 (22415): BOINC client no longer exists - exiting
11:22:35 (22415): timer handler: client dead, exiting
11:22:40 (22415): BOINC client no longer exists - exiting
11:22:40 (22415): timer handler: client dead, exiting
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0078
(argv4) batchid: 968
(argv5) wuid: 12184722
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/sdc/home/seti/BOINC/slots/1
Project directory is: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722
..mkdir for temp folder for results failed
Copying: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_02e24f44fd70c3771f6dcc569c803354 to: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_0078_1999050100_123_968_12184722.zip
Unzipping the namelist zip file: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_0078_1999050100_123_968_12184722.zip
ic_ancil_file: ic_ancil_12184722
ifsdata_file: ifsdata_12184722
climate_data_file: clim_data_12184722
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_add0d6b2d2d22ca2a8c217229ba33bff to: /mnt/sdc/home/seti/BOINC/slots/1/ic_ancil_12184722.zip
Unzipping the IC ancils zip file: /mnt/sdc/home/seti/BOINC/slots/1/ic_ancil_12184722.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd38a83af2e48cefacc5d784e0c27518 to: /mnt/sdc/home/seti/BOINC/slots/1/ifsdata/ifsdata_12184722.zip
Unzipping the ifsdata_zip file: /mnt/sdc/home/seti/BOINC/slots/1/ifsdata/ifsdata_12184722.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/sdc/home/seti/BOINC/slots/1/159l_2/clim_data_12184722.zip
Unzipping the climate data zip file: /mnt/sdc/home/seti/BOINC/slots/1/159l_2/clim_data_12184722.zip
Checking for progress XML file: /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
Opened progress file ok : /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
-- Model is restarting --
Adjusting last_iter, 1020, to previous model restart step.
Creating progress file: /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
last_cpu_time: 22983
upload_file_number: 42
last_iter: 1007
last_upload: 3628800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484
The child process has been launched with process id: 4879
Executing the command: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:4879:4879] [20230105:113438:1672914878:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
11:35:24 STEP 1008 H=1008:00 +CPU= 31.703
11:35:42 STEP 1009 H=1009:00 +CPU= 17.147
11:36:01 STEP 1010 H=1010:00 +CPU= 18.847
11:36:28 STEP 1011 H=1011:00 +CPU= 27.416
11:36:47 STEP 1012 H=1012:00 +CPU= 18.439
11:37:04 STEP 1013 H=1013:00 +CPU= 16.951
11:37:31 STEP 1014 H=1014:00 +CPU= 27.631
11:37:49 STEP 1015 H=1015:00 +CPU= 17.218
11:38:08 STEP 1016 H=1016:00 +CPU= 19.153
11:38:36 STEP 1017 H=1017:00 +CPU= 28.421
11:38:54 STEP 1018 H=1018:00 +CPU= 17.849
11:39:11 STEP 1019 H=1019:00 +CPU= 16.939
11:39:44 STEP 1020 H=1020:00 +CPU= 32.092
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001020
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001020
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001020
11:40:01 STEP 1021 H=1021:00 +CPU= 17.626
11:40:19 STEP 1022 H=1022:00 +CPU= 18.073
11:40:47 STEP 1023 H=1023:00 +CPU= 27.536
11:41:06 STEP 1024 H=1024:00 +CPU= 18.403
11:41:23 STEP 1025 H=1025:00 +CPU= 17.448
11:41:52 STEP 1026 H=1026:00 +CPU= 28.713
11:42:09 STEP 1027 H=1027:00 +CPU= 17.358
11:42:27 STEP 1028 H=1028:00 +CPU= 17.337
11:42:54 STEP 1029 H=1029:00 +CPU= 26.842
11:43:11 STEP 1030 H=1030:00 +CPU= 17.334
11:43:29 STEP 1031 H=1031:00 +CPU= 18.227
11:44:01 STEP 1032 H=1032:00 +CPU= 31.786
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001032
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001032
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001032
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001008
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001008
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001008
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001020
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001020
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001020
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_42.zip
Uploading the intermediate file: upload_file_42.zip
11:44:20 STEP 1033 H=1033:00 +CPU= 18.416
11:44:38 STEP 1034 H=1034:00 +CPU= 18.356
11:45:07 STEP 1035 H=1035:00 +CPU= 28.181
11:45:25 STEP 1036 H=1036:00 +CPU= 18.128
11:45:44 STEP 1037 H=1037:00 +CPU= 18.943
11:46:12 STEP 1038 H=1038:00 +CPU= 28.142
11:46:31 STEP 1039 H=1039:00 +CPU= 18.642
11:46:50 STEP 1040 H=1040:00 +CPU= 18.983
11:47:18 STEP 1041 H=1041:00 +CPU= 27.595
11:47:35 STEP 1042 H=1042:00 +CPU= 17.477
11:47:53 STEP 1043 H=1043:00 +CPU= 17.459
11:48:23 STEP 1044 H=1044:00 +CPU= 29.982
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001044
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001044
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001044
11:48:41 STEP 1045 H=1045:00 +CPU= 17.854
11:48:59 STEP 1046 H=1046:00 +CPU= 18.232
11:49:28 STEP 1047 H=1047:00 +CPU= 28.384
11:49:45 STEP 1048 H=1048:00 +CPU= 17.634
11:50:04 STEP 1049 H=1049:00 +CPU= 19.103
11:50:34 STEP 1050 H=1050:00 +CPU= 28.875
11:50:50 STEP 1051 H=1051:00 +CPU= 16.680
11:51:08 STEP 1052 H=1052:00 +CPU= 17.983
11:51:37 STEP 1053 H=1053:00 +CPU= 28.201
11:51:55 STEP 1054 H=1054:00 +CPU= 18.325
11:52:13 STEP 1055 H=1055:00 +CPU= 17.785
11:52:46 STEP 1056 H=1056:00 +CPU= 33.136
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001056
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001056
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001056
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001032
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001032
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001032
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001044
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001044
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001044
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_43.zip
Uploading the intermediate file: upload_file_43.zip
11:53:03 STEP 1057 H=1057:00 +CPU= 16.733
11:53:20 STEP 1058 H=1058:00 +CPU= 17.275
11:53:51 STEP 1059 H=1059:00 +CPU= 30.562
11:54:08 STEP 1060 H=1060:00 +CPU= 16.914
11:54:24 STEP 1061 H=1061:00 +CPU= 16.334
11:54:53 STEP 1062 H=1062:00 +CPU= 28.605
11:55:10 STEP 1063 H=1063:00 +CPU= 16.995
11:55:28 STEP 1064 H=1064:00 +CPU= 17.755
11:55:56 STEP 1065 H=1065:00 +CPU= 28.153
11:56:16 STEP 1066 H=1066:00 +CPU= 19.311
11:56:33 STEP 1067 H=1067:00 +CPU= 17.326
11:57:04 STEP 1068 H=1068:00 +CPU= 30.358
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001068
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001068
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001068
11:57:20 STEP 1069 H=1069:00 +CPU= 16.853
11:57:39 STEP 1070 H=1070:00 +CPU= 18.050
11:58:07 STEP 1071 H=1071:00 +CPU= 28.099
11:58:24 STEP 1072 H=1072:00 +CPU= 17.248
11:58:44 STEP 1073 H=1073:00 +CPU= 19.541
11:59:13 STEP 1074 H=1074:00 +CPU= 29.273
11:59:31 STEP 1075 H=1075:00 +CPU= 17.858
11:59:49 STEP 1076 H=1076:00 +CPU= 17.844
12:00:17 STEP 1077 H=1077:00 +CPU= 28.281
12:00:35 STEP 1078 H=1078:00 +CPU= 17.921
12:00:55 STEP 1079 H=1079:00 +CPU= 19.480
12:01:26 STEP 1080 H=1080:00 +CPU= 30.834
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001080
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001080
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001080
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001056
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001056
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001056
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001068
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001068
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001068
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_44.zip
Uploading the intermediate file: upload_file_44.zip
12:01:43 STEP 1081 H=1081:00 +CPU= 17.033
12:02:02 STEP 1082 H=1082:00 +CPU= 18.541
12:02:30 STEP 1083 H=1083:00 +CPU= 28.286
12:02:48 STEP 1084 H=1084:00 +CPU= 18.223
12:03:06 STEP 1085 H=1085:00 +CPU= 17.909
12:03:35 STEP 1086 H=1086:00 +CPU= 28.769
12:03:52 STEP 1087 H=1087:00 +CPU= 16.780
12:04:11 STEP 1088 H=1088:00 +CPU= 18.507
12:04:39 STEP 1089 H=1089:00 +CPU= 28.501
12:04:57 STEP 1090 H=1090:00 +CPU= 17.901
12:05:15 STEP 1091 H=1091:00 +CPU= 17.756
12:05:48 STEP 1092 H=1092:00 +CPU= 32.921
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001092
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001092
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001092
12:06:07 STEP 1093 H=1093:00 +CPU= 18.736
12:06:26 STEP 1094 H=1094:00 +CPU= 18.881
12:06:55 STEP 1095 H=1095:00 +CPU= 28.572
12:07:13 STEP 1096 H=1096:00 +CPU= 18.222
12:07:31 STEP 1097 H=1097:00 +CPU= 17.981
12:08:00 STEP 1098 H=1098:00 +CPU= 29.100
12:08:17 STEP 1099 H=1099:00 +CPU= 17.062
12:08:37 STEP 1100 H=1100:00 +CPU= 19.327
12:09:05 STEP 1101 H=1101:00 +CPU= 27.927
12:09:24 STEP 1102 H=1102:00 +CPU= 18.756
12:09:43 STEP 1103 H=1103:00 +CPU= 18.807
12:10:16 STEP 1104 H=1104:00 +CPU= 33.266
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001104
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001104
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001104
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001080
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001080
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001080
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001092
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001092
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001092
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_45.zip
Uploading the intermediate file: upload_file_45.zip
12:10:34 STEP 1105 H=1105:00 +CPU= 17.812
12:10:52 STEP 1106 H=1106:00 +CPU= 18.230
12:11:21 STEP 1107 H=1107:00 +CPU= 28.457
12:11:40 STEP 1108 H=1108:00 +CPU= 18.559
12:11:58 STEP 1109 H=1109:00 +CPU= 18.397
12:12:26 STEP 1110 H=1110:00 +CPU= 28.236
12:12:43 STEP 1111 H=1111:00 +CPU= 16.275
12:13:00 STEP 1112 H=1112:00 +CPU= 17.003
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:22:30 STEP 1113 H=1113:00 +CPU= 26.525
12:22:47 STEP 1114 H=1114:00 +CPU= 16.807
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:43:40 STEP 1115 H=1115:00 +CPU= 16.543
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0078
(argv4) batchid: 968
(argv5) wuid: 12184722
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/sdc/home/seti/BOINC/slots/1
Project directory is: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722
..mkdir for temp folder for results failed
Copying: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_02e24f44fd70c3771f6dcc569c803354 to: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_0078_1999050100_123_968_12184722.zip
Unzipping the namelist zip file: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_0078_1999050100_123_968_12184722.zip
ic_ancil_file: ic_ancil_12184722
ifsdata_file: ifsdata_12184722
climate_data_file: clim_data_12184722
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_add0d6b2d2d22ca2a8c217229ba33bff to: /mnt/sdc/home/seti/BOINC/slots/1/ic_ancil_12184722.zip
Unzipping the IC ancils zip file: /mnt/sdc/home/seti/BOINC/slots/1/ic_ancil_12184722.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd38a83af2e48cefacc5d784e0c27518 to: /mnt/sdc/home/seti/BOINC/slots/1/ifsdata/ifsdata_12184722.zip
Unzipping the ifsdata_zip file: /mnt/sdc/home/seti/BOINC/slots/1/ifsdata/ifsdata_12184722.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/sdc/home/seti/BOINC/slots/1/159l_2/clim_data_12184722.zip
Unzipping the climate data zip file: /mnt/sdc/home/seti/BOINC/slots/1/159l_2/clim_data_12184722.zip
Checking for progress XML file: /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
Opened progress file ok : /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
-- Model is restarting --
Adjusting last_iter, 1116, to previous model restart step.
Creating progress file: /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
last_cpu_time: 25323
upload_file_number: 46
last_iter: 1103
last_upload: 3974400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484
The child process has been launched with process id: 9322
Executing the command: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:9322:9322] [20230106:091814:1672993094:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
09:18:58 STEP 1104 H=1104:00 +CPU= 29.729
09:19:15 STEP 1105 H=1105:00 +CPU= 16.521
09:19:32 STEP 1106 H=1106:00 +CPU= 17.151
09:19:59 STEP 1107 H=1107:00 +CPU= 26.472
09:20:15 STEP 1108 H=1108:00 +CPU= 16.856
09:20:32 STEP 1109 H=1109:00 +CPU= 16.556
09:20:59 STEP 1110 H=1110:00 +CPU= 27.197
09:21:16 STEP 1111 H=1111:00 +CPU= 16.283
09:21:33 STEP 1112 H=1112:00 +CPU= 16.818
09:21:59 STEP 1113 H=1113:00 +CPU= 26.560
09:22:16 STEP 1114 H=1114:00 +CPU= 16.917
09:22:33 STEP 1115 H=1115:00 +CPU= 16.366
09:23:02 STEP 1116 H=1116:00 +CPU= 29.240
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001116
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001116
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001116
09:23:18 STEP 1117 H=1117:00 +CPU= 16.276
09:23:35 STEP 1118 H=1118:00 +CPU= 16.897
09:24:02 STEP 1119 H=1119:00 +CPU= 26.534
09:24:19 STEP 1120 H=1120:00 +CPU= 17.435
09:24:36 STEP 1121 H=1121:00 +CPU= 16.410
09:25:03 STEP 1122 H=1122:00 +CPU= 27.202
09:25:20 STEP 1123 H=1123:00 +CPU= 16.488
09:25:37 STEP 1124 H=1124:00 +CPU= 16.831
09:26:03 STEP 1125 H=1125:00 +CPU= 26.467
09:26:20 STEP 1126 H=1126:00 +CPU= 16.956
09:26:37 STEP 1127 H=1127:00 +CPU= 16.744
09:27:08 STEP 1128 H=1128:00 +CPU= 30.401
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001128
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001128
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001128
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001104
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001104
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001104
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001116
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001116
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001116
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_46.zip
09:27:24 STEP 1129 H=1129:00 +CPU= 16.438
Uploading the intermediate file: upload_file_46.zip
09:27:41 STEP 1130 H=1130:00 +CPU= 17.071
09:28:08 STEP 1131 H=1131:00 +CPU= 26.549
09:28:25 STEP 1132 H=1132:00 +CPU= 17.043
09:28:41 STEP 1133 H=1133:00 +CPU= 16.319
09:29:09 STEP 1134 H=1134:00 +CPU= 27.222
09:29:25 STEP 1135 H=1135:00 +CPU= 16.449
09:29:42 STEP 1136 H=1136:00 +CPU= 16.805
09:30:08 STEP 1137 H=1137:00 +CPU= 26.482
09:30:25 STEP 1138 H=1138:00 +CPU= 16.986
09:30:42 STEP 1139 H=1139:00 +CPU= 16.487
09:31:11 STEP 1140 H=1140:00 +CPU= 29.256
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001140
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001140
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001140
09:31:28 STEP 1141 H=1141:00 +CPU= 16.385
09:31:45 STEP 1142 H=1142:00 +CPU= 17.121
09:32:12 STEP 1143 H=1143:00 +CPU= 26.543
09:32:29 STEP 1144 H=1144:00 +CPU= 17.041
09:32:45 STEP 1145 H=1145:00 +CPU= 16.714
09:33:13 STEP 1146 H=1146:00 +CPU= 27.269
09:33:30 STEP 1147 H=1147:00 +CPU= 16.891
09:33:47 STEP 1148 H=1148:00 +CPU= 17.346
09:34:14 STEP 1149 H=1149:00 +CPU= 26.489
09:34:31 STEP 1150 H=1150:00 +CPU= 17.131
09:34:48 STEP 1151 H=1151:00 +CPU= 16.914
09:35:19 STEP 1152 H=1152:00 +CPU= 30.865
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001152
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001152
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001152
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001128
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001128
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001128
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001140
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001140
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001140
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_47.zip
Uploading the intermediate file: upload_file_47.zip
09:35:35 STEP 1153 H=1153:00 +CPU= 16.392
09:35:52 STEP 1154 H=1154:00 +CPU= 16.959
09:36:19 STEP 1155 H=1155:00 +CPU= 26.400
09:36:36 STEP 1156 H=1156:00 +CPU= 16.971
09:36:52 STEP 1157 H=1157:00 +CPU= 16.461
09:37:20 STEP 1158 H=1158:00 +CPU= 27.370
09:37:36 STEP 1159 H=1159:00 +CPU= 16.249
09:37:54 STEP 1160 H=1160:00 +CPU= 17.255
09:38:20 STEP 1161 H=1161:00 +CPU= 26.707
09:38:37 STEP 1162 H=1162:00 +CPU= 16.973
09:38:54 STEP 1163 H=1163:00 +CPU= 16.522
09:39:23 STEP 1164 H=1164:00 +CPU= 29.235
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001164
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001164
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001164
09:39:40 STEP 1165 H=1165:00 +CPU= 16.368
09:39:57 STEP 1166 H=1166:00 +CPU= 16.939
09:40:23 STEP 1167 H=1167:00 +CPU= 26.331
09:40:40 STEP 1168 H=1168:00 +CPU= 16.930
09:40:57 STEP 1169 H=1169:00 +CPU= 16.368
09:41:24 STEP 1170 H=1170:00 +CPU= 27.039
09:41:40 STEP 1171 H=1171:00 +CPU= 16.297
09:41:57 STEP 1172 H=1172:00 +CPU= 16.845
09:42:23 STEP 1173 H=1173:00 +CPU= 26.338
09:42:40 STEP 1174 H=1174:00 +CPU= 16.983
09:42:57 STEP 1175 H=1175:00 +CPU= 16.708
09:43:27 STEP 1176 H=1176:00 +CPU= 30.259
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001176
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001176
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001176
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001152
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001152
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001152
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001164
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001164
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001164
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_48.zip
Uploading the intermediate file: upload_file_48.zip
09:43:44 STEP 1177 H=1177:00 +CPU= 16.382
09:44:01 STEP 1178 H=1178:00 +CPU= 16.758
09:44:27 STEP 1179 H=1179:00 +CPU= 26.353
09:44:44 STEP 1180 H=1180:00 +CPU= 16.882
09:45:00 STEP 1181 H=1181:00 +CPU= 16.159
09:45:27 STEP 1182 H=1182:00 +CPU= 27.094
09:45:44 STEP 1183 H=1183:00 +CPU= 16.309
09:46:01 STEP 1184 H=1184:00 +CPU= 16.974
09:46:27 STEP 1185 H=1185:00 +CPU= 26.384
09:46:45 STEP 1186 H=1186:00 +CPU= 17.193
09:47:01 STEP 1187 H=1187:00 +CPU= 16.520
09:47:30 STEP 1188 H=1188:00 +CPU= 29.173
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001188
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001188
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001188
09:47:47 STEP 1189 H=1189:00 +CPU= 16.314
09:48:04 STEP 1190 H=1190:00 +CPU= 16.852
09:48:30 STEP 1191 H=1191:00 +CPU= 26.428
09:48:47 STEP 1192 H=1192:00 +CPU= 16.885
09:49:03 STEP 1193 H=1193:00 +CPU= 16.203
09:49:30 STEP 1194 H=1194:00 +CPU= 27.091
09:49:47 STEP 1195 H=1195:00 +CPU= 16.304
09:50:04 STEP 1196 H=1196:00 +CPU= 16.836
09:50:30 STEP 1197 H=1197:00 +CPU= 26.460
09:50:47 STEP 1198 H=1198:00 +CPU= 16.868
09:51:04 STEP 1199 H=1199:00 +CPU= 16.772
09:51:34 STEP 1200 H=1200:00 +CPU= 30.339
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001200
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001200
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001200
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001176
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001176
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001176
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001188
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001188
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001188
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_49.zip
Uploading the intermediate file: upload_file_49.zip
09:51:51 STEP 1201 H=1201:00 +CPU= 16.434
09:52:08 STEP 1202 H=1202:00 +CPU= 16.846
09:52:34 STEP 1203 H=1203:00 +CPU= 26.497
09:52:52 STEP 1204 H=1204:00 +CPU= 17.183
09:53:08 STEP 1205 H=1205:00 +CPU= 16.390
09:53:35 STEP 1206 H=1206:00 +CPU= 26.994
09:53:52 STEP 1207 H=1207:00 +CPU= 16.394
09:54:09 STEP 1208 H=1208:00 +CPU= 17.036
09:54:35 STEP 1209 H=1209:00 +CPU= 26.457
09:54:52 STEP 1210 H=1210:00 +CPU= 16.865
09:55:08 STEP 1211 H=1211:00 +CPU= 16.347
09:55:38 STEP 1212 H=1212:00 +CPU= 29.246
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001212
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001212
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001212
09:55:54 STEP 1213 H=1213:00 +CPU= 16.316
09:56:11 STEP 1214 H=1214:00 +CPU= 16.973
09:56:38 STEP 1215 H=1215:00 +CPU= 26.450
09:56:55 STEP 1216 H=1216:00 +CPU= 16.971
09:57:11 STEP 1217 H=1217:00 +CPU= 16.299
09:57:38 STEP 1218 H=1218:00 +CPU= 27.069
09:57:54 STEP 1219 H=1219:00 +CPU= 16.280
09:58:11 STEP 1220 H=1220:00 +CPU= 16.823
09:58:38 STEP 1221 H=1221:00 +CPU= 26.495
09:58:55 STEP 1222 H=1222:00 +CPU= 16.785
09:59:12 STEP 1223 H=1223:00 +CPU= 17.021
09:59:43 STEP 1224 H=1224:00 +CPU= 30.947
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001224
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001224
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001224
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001200
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001200
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001200
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001212
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001212
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001212
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_50.zip
Uploading the intermediate file: upload_file_50.zip
09:59:59 STEP 1225 H=1225:00 +CPU= 16.425
10:00:16 STEP 1226 H=1226:00 +CPU= 16.800
10:00:43 STEP 1227 H=1227:00 +CPU= 26.498
10:01:00 STEP 1228 H=1228:00 +CPU= 16.903
10:01:16 STEP 1229 H=1229:00 +CPU= 16.323
10:01:43 STEP 1230 H=1230:00 +CPU= 27.091
10:02:00 STEP 1231 H=1231:00 +CPU= 16.526
10:02:17 STEP 1232 H=1232:00 +CPU= 16.889
10:02:44 STEP 1233 H=1233:00 +CPU= 26.966
10:03:01 STEP 1234 H=1234:00 +CPU= 17.092
10:03:18 STEP 1235 H=1235:00 +CPU= 16.671
10:03:47 STEP 1236 H=1236:00 +CPU= 29.375
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001236
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001236
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001236
10:04:04 STEP 1237 H=1237:00 +CPU= 16.561
10:04:21 STEP 1238 H=1238:00 +CPU= 16.853
10:04:47 STEP 1239 H=1239:00 +CPU= 26.592
10:05:05 STEP 1240 H=1240:00 +CPU= 17.403
10:05:21 STEP 1241 H=1241:00 +CPU= 16.341
10:05:49 STEP 1242 H=1242:00 +CPU= 27.153
10:06:05 STEP 1243 H=1243:00 +CPU= 16.569
10:06:23 STEP 1244 H=1244:00 +CPU= 17.321
10:06:49 STEP 1245 H=1245:00 +CPU= 26.580
10:07:06 STEP 1246 H=1246:00 +CPU= 16.922
10:07:23 STEP 1247 H=1247:00 +CPU= 16.888
10:07:54 STEP 1248 H=1248:00 +CPU= 30.491
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001248
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001248
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001248
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001224
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001224
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001224
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001236
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001236
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001236
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_51.zip
Uploading the intermediate file: upload_file_51.zip
10:08:10 STEP 1249 H=1249:00 +CPU= 16.303
10:08:27 STEP 1250 H=1250:00 +CPU= 16.843
10:08:54 STEP 1251 H=1251:00 +CPU= 26.553
10:09:11 STEP 1252 H=1252:00 +CPU= 16.926
10:09:27 STEP 1253 H=1253:00 +CPU= 16.224
10:09:54 STEP 1254 H=1254:00 +CPU= 27.120
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:45:16 STEP 1255 H=1255:00 +CPU= 16.288
10:45:32 STEP 1256 H=1256:00 +CPU= 16.721
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
11:04:52 (9302): BOINC client no longer exists - exiting
11:04:52 (9302): timer handler: client dead, exiting
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0078
(argv4) batchid: 968
(argv5) wuid: 12184722
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/sdc/home/seti/BOINC/slots/1
Project directory is: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722
..mkdir for temp folder for results failed
Copying: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_02e24f44fd70c3771f6dcc569c803354 to: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_0078_1999050100_123_968_12184722.zip
Unzipping the namelist zip file: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_ps_0078_1999050100_123_968_12184722.zip
ic_ancil_file: ic_ancil_12184722
ifsdata_file: ifsdata_12184722
climate_data_file: clim_data_12184722
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_add0d6b2d2d22ca2a8c217229ba33bff to: /mnt/sdc/home/seti/BOINC/slots/1/ic_ancil_12184722.zip
Unzipping the IC ancils zip file: /mnt/sdc/home/seti/BOINC/slots/1/ic_ancil_12184722.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_cd38a83af2e48cefacc5d784e0c27518 to: /mnt/sdc/home/seti/BOINC/slots/1/ifsdata/ifsdata_12184722.zip
Unzipping the ifsdata_zip file: /mnt/sdc/home/seti/BOINC/slots/1/ifsdata/ifsdata_12184722.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/sdc/home/seti/BOINC/slots/1/159l_2/clim_data_12184722.zip
Unzipping the climate data zip file: /mnt/sdc/home/seti/BOINC/slots/1/159l_2/clim_data_12184722.zip
Checking for progress XML file: /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
Opened progress file ok : /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
-- Model is restarting --
Adjusting last_iter, 1257, to previous model restart step.
Creating progress file: /mnt/sdc/home/seti/BOINC/slots/1/progress_file_12184722.xml
last_cpu_time: 28438
upload_file_number: 52
last_iter: 1247
last_upload: 4492800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484
The child process has been launched with process id: 12509
Executing the command: /mnt/sdc/home/seti/BOINC/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:merkur:1:1:12509:12509] [20230106:112838:1673000918:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
11:29:08 STEP 1248 H=1248:00 +CPU= 29.163
11:29:25 STEP 1249 H=1249:00 +CPU= 16.578
11:29:44 STEP 1250 H=1250:00 +CPU= 18.414
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:34:31 STEP 1251 H=1251:00 +CPU= 29.650
11:34:49 STEP 1252 H=1252:00 +CPU= 18.852
11:35:08 STEP 1253 H=1253:00 +CPU= 18.399
11:35:38 STEP 1254 H=1254:00 +CPU= 30.411
11:35:57 STEP 1255 H=1255:00 +CPU= 17.961
11:36:15 STEP 1256 H=1256:00 +CPU= 17.960
11:36:43 STEP 1257 H=1257:00 +CPU= 27.754
11:37:02 STEP 1258 H=1258:00 +CPU= 19.365
11:37:19 STEP 1259 H=1259:00 +CPU= 16.851
11:37:51 STEP 1260 H=1260:00 +CPU= 32.038
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001260
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001260
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001260
11:38:08 STEP 1261 H=1261:00 +CPU= 17.151
11:38:26 STEP 1262 H=1262:00 +CPU= 17.472
11:38:54 STEP 1263 H=1263:00 +CPU= 28.364
11:39:14 STEP 1264 H=1264:00 +CPU= 19.643
11:39:31 STEP 1265 H=1265:00 +CPU= 17.354
11:40:02 STEP 1266 H=1266:00 +CPU= 29.993
11:40:20 STEP 1267 H=1267:00 +CPU= 18.659
11:40:38 STEP 1268 H=1268:00 +CPU= 17.937
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:41:28 STEP 1269 H=1269:00 +CPU= 28.491
11:41:46 STEP 1270 H=1270:00 +CPU= 18.105
11:42:05 STEP 1271 H=1271:00 +CPU= 18.869
11:42:38 STEP 1272 H=1272:00 +CPU= 32.954
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMGGhq0f+001272
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMSHhq0f+001272
Moving to projects directory: /mnt/sdc/home/seti/BOINC/slots/1/ICMUAhq0f+001272
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001248
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001248
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001248
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMGGhq0f+001260
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMSHhq0f+001260
Adding to the zip: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_12184722/ICMUAhq0f+001260
Zipping up the intermediate file: /mnt/sdc/home/seti/BOINC/projects/climateprediction.net/oifs_43r3_ps_0078_1999050100_123_968_12184722_0_r1695325484_52.zip
Uploading the intermediate file: upload_file_52.zip
11:42:56 STEP 1273 H=1273:00 +CPU= 17.523
11:43:14 STEP 1274 H=1274:00 +CPU= 17.590
11:43:42 STEP 1275 H=1275:00 +CPU= 28.477
11:44:00 STEP 1276 H=1276:00 +CPU= 18.013
11:44:18 STEP 1277 H=1277:00 +CPU= 17.365
11:44:49 STEP 1278 H=1278:00 +CPU= 30.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:15:07 STEP 1279 H=1279:00 +CPU= 17.039
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
