Name | oifs_43r3_ps_0007_2002050100_123_971_12187651_0 |
Workunit | 12187651 |
Created | 23 Dec 2022, 1:35:51 UTC |
Sent | 3 Jan 2023, 16:20:43 UTC |
Report deadline | 2 Feb 2023, 16:20:43 UTC |
Received | 8 Jan 2023, 8:05:55 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1534746 |
Run time | 2 hours 19 min 9 sec |
CPU time | 2 hours 19 min 9 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 5.61 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,399.70 MB |
Peak swap size | 4,974.32 MB |
Peak disk usage | 1,296.87 MB |
Stderr | <core_client_version>7.20.5</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> g directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip 09:54:43 STEP 32 H= 32:00 +CPU= 44.552 09:54:47 STEP 27 H= 27:00 +CPU= 56.041 13:31:47 STEP 28 H= 28:00 +CPU=172.035 13:31:53 STEP 33 H= 33:00 +CPU=186.946 13:32:05 STEP 29 H= 29:00 +CPU= 17.328 Copying the namelist files from: ../../projects/climateprediction.net/jf_046da7b448f5e66dcddbba1c5a43b092 to: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip Unzipping the namelist zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip ic_ancil_file: ic_ancil_12187651 ifsdata_file: ifsdata_12187651 climate_data_file: clim_data_12187651 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_bf47387d470b76f2ece6d6b63c2b8e18 to: /var/lib/boinc/slots/24/ic_ancil_12187651.zip 13:56:16 STEP 34 H= 34:00 +CPU= 24.675 Unzipping the IC ancils zip file: /var/lib/boinc/slots/24/ic_ancil_12187651.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f9597ab633dd9e416d1c20f12cbde2d to: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip 15:28:57 STEP 35 H= 35:00 +CPU= 35.096 15:29:03 STEP 30 H= 30:00 +CPU= 56.010 Copying the namelist files from: ../../projects/climateprediction.net/jf_046da7b448f5e66dcddbba1c5a43b092 to: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip Unzipping the namelist zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip ic_ancil_file: ic_ancil_12187651 ifsdata_file: ifsdata_12187651 climate_data_file: clim_data_12187651 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_bf47387d470b76f2ece6d6b63c2b8e18 to: /var/lib/boinc/slots/24/ic_ancil_12187651.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/24/ic_ancil_12187651.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f9597ab633dd9e416d1c20f12cbde2d to: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Unzipping the climate data zip file: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip 15:29:25 STEP 31 H= 31:00 +CPU= 20.400 Checking for progress XML file: /var/lib/boinc/slots/24/progress_file_12187651.xml Opened progress file ok : /var/lib/boinc/slots/24/progress_file_12187651.xml -- Model is restarting -- Adjusting last_iter, -985, to previous model restart step. Creating progress file: /var/lib/boinc/slots/24/progress_file_12187651.xml last_cpu_time: 7886 upload_file_number: 1 last_iter: -985 last_upload: 86400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0007_2002050100_123_971_12187651_0_r984622111 The child process has been launched with process id: 8718 Executing the command: /var/lib/boinc/slots/24/oifs_43r3_model.exe 15:29:32 STEP 36 H= 36:00 +CPU= 31.900 [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.183] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.188] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.191] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.191] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.191] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:8718:8718] [20230107:152933:1673080173:0.191] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:31:42 STEP 32 H= 32:00 +CPU= 32.525 Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = 15:31:49 STEP 37 H= 37:00 +CPU= 31.846 Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_046da7b448f5e66dcddbba1c5a43b092 to: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip Unzipping the namelist zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip ic_ancil_file: ic_ancil_12187651 ifsdata_file: ifsdata_12187651 climate_data_file: clim_data_12187651 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_bf47387d470b76f2ece6d6b63c2b8e18 to: /var/lib/boinc/slots/24/ic_ancil_12187651.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/24/ic_ancil_12187651.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f9597ab633dd9e416d1c20f12cbde2d to: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Unzipping the climate data zip file: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Checking for progress XML file: /var/lib/boinc/slots/24/progress_file_12187651.xml Opened progress file ok : /var/lib/boinc/slots/24/progress_file_12187651.xml -- Model is restarting -- Adjusting last_iter, -999, to previous model restart step. Creating progress file: /var/lib/boinc/slots/24/progress_file_12187651.xml last_cpu_time: 7909 upload_file_number: 1 last_iter: -985 last_upload: 86400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0007_2002050100_123_971_12187651_0_r984622111 The child process has been launched with process id: 9138 Executing the command: /var/lib/boinc/slots/24/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9138:9138] [20230107:193909:1673095149:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:39:09 STEP 24 H= 24:00 +CPU=262.066 19:39:28 STEP 25 H= 25:00 +CPU= 16.285 Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_046da7b448f5e66dcddbba1c5a43b092 to: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip Unzipping the namelist zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip ic_ancil_file: ic_ancil_12187651 ifsdata_file: ifsdata_12187651 climate_data_file: clim_data_12187651 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_bf47387d470b76f2ece6d6b63c2b8e18 to: /var/lib/boinc/slots/24/ic_ancil_12187651.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/24/ic_ancil_12187651.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f9597ab633dd9e416d1c20f12cbde2d to: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Unzipping the climate data zip file: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Checking for progress XML file: /var/lib/boinc/slots/24/progress_file_12187651.xml Opened progress file ok : /var/lib/boinc/slots/24/progress_file_12187651.xml -- Model is restarting -- Adjusting last_iter, -999, to previous model restart step. Creating progress file: /var/lib/boinc/slots/24/progress_file_12187651.xml last_cpu_time: 8008 upload_file_number: 1 last_iter: -985 last_upload: 86400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0007_2002050100_123_971_12187651_0_r984622111 The child process has been launched with process id: 9780 Executing the command: /var/lib/boinc/slots/24/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:9780:9780] [20230108:003154:1673112714:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process 00:32:00 STEP 24 H= 24:00 +CPU=399.422 Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_046da7b448f5e66dcddbba1c5a43b092 to: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip Unzipping the namelist zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip ic_ancil_file: ic_ancil_12187651 ifsdata_file: ifsdata_12187651 climate_data_file: clim_data_12187651 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_bf47387d470b76f2ece6d6b63c2b8e18 to: /var/lib/boinc/slots/24/ic_ancil_12187651.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/24/ic_ancil_12187651.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f9597ab633dd9e416d1c20f12cbde2d to: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Unzipping the climate data zip file: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Checking for progress XML file: /var/lib/boinc/slots/24/progress_file_12187651.xml Opened progress file ok : /var/lib/boinc/slots/24/progress_file_12187651.xml -- Model is restarting -- Adjusting last_iter, -999, to previous model restart step. Creating progress file: /var/lib/boinc/slots/24/progress_file_12187651.xml last_cpu_time: 8030 upload_file_number: 1 last_iter: -985 last_upload: 86400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0007_2002050100_123_971_12187651_0_r984622111 The child process has been launched with process id: 10439 Executing the command: /var/lib/boinc/slots/24/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10439:10439] [20230108:085130:1673142690:0.048] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0007 (argv4) batchid: 971 (argv5) wuid: 12187651 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc/slots/24 Project directory is: /var/lib/boinc/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187651 ..mkdir for temp folder for results failed Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_046da7b448f5e66dcddbba1c5a43b092 to: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip Unzipping the namelist zip file: /var/lib/boinc/slots/24/oifs_43r3_ps_0007_2002050100_123_971_12187651.zip ic_ancil_file: ic_ancil_12187651 ifsdata_file: ifsdata_12187651 climate_data_file: clim_data_12187651 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_bf47387d470b76f2ece6d6b63c2b8e18 to: /var/lib/boinc/slots/24/ic_ancil_12187651.zip Unzipping the IC ancils zip file: /var/lib/boinc/slots/24/ic_ancil_12187651.zip 10:34:02 STEP 24 H= 24:00 +CPU=229.642 ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1f9597ab633dd9e416d1c20f12cbde2d to: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip Unzipping the ifsdata_zip file: /var/lib/boinc/slots/24/ifsdata/ifsdata_12187651.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Unzipping the climate data zip file: /var/lib/boinc/slots/24/159l_2/clim_data_12187651.zip Checking for progress XML file: /var/lib/boinc/slots/24/progress_file_12187651.xml Opened progress file ok : /var/lib/boinc/slots/24/progress_file_12187651.xml -- Model is restarting -- Adjusting last_iter, 24, to previous model restart step. Creating progress file: /var/lib/boinc/slots/24/progress_file_12187651.xml last_cpu_time: 8099 upload_file_number: 1 last_iter: 23 last_upload: 86400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0007_2002050100_123_971_12187651_0_r984622111 The child process has been launched with process id: 10591 Executing the command: /var/lib/boinc/slots/24/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:vm9:1:1:10591:10591] [20230108:103405:1673148845:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 10:34:27 STEP 25 H= 25:00 +CPU= 16.844 Invalid characters in stoi string: STEPO Unable to update iter, resetting to last_iter. ifs_line = Suspend request received from the BOINC client, suspending the child process Resuming the child process 10:34:44 STEP 26 H= 26:00 +CPU= 15.889 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:10:20 STEP 24 H= 24:00 +CPU= 83.853 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:10:39 STEP 25 H= 25:00 +CPU= 14.977 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process The child process terminated with status: 0 ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 climateprediction.net