Name | oifs_43r3_ps_0732_2020050100_123_989_12206376_1 |
Workunit | 12206376 |
Created | 26 Dec 2022, 13:33:31 UTC |
Sent | 16 Jan 2023, 21:59:25 UTC |
Report deadline | 17 Mar 2023, 21:59:25 UTC |
Received | 22 Jan 2023, 10:46:54 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1538188 |
Run time | 7 hours 47 min 17 sec |
CPU time | 2 min 3 sec |
Validate state | Invalid |
Credit | 7,257.60 |
Device peak FLOPS | 9.06 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,407.50 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,185.07 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> ile: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip ic_ancil_file: ic_ancil_12206376 ifsdata_file: ifsdata_12206376 climate_data_file: clim_data_12206376 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_cf69bd1ab6ca2467bfd1a3f83c2e0dc0 to: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c49648ee49e9f213b54d78d67bc69f8 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206376.xml -- Model is restarting -- Adjusting last_iter, 1516, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml last_cpu_time: 5687 upload_file_number: 23 last_iter: 1511 last_upload: 5443200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0732_2020050100_123_989_12206376_1_r187916441 The child process has been launched with process id: 1343159 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1343159:1343159] [20230122:094112:1674373272:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0732 (argv4) batchid: 989 (argv5) wuid: 12206376 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206376 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_734e4ce79bcf9d75a687480617c6b9bd to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip ic_ancil_file: ic_ancil_12206376 ifsdata_file: ifsdata_12206376 climate_data_file: clim_data_12206376 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_cf69bd1ab6ca2467bfd1a3f83c2e0dc0 to: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c49648ee49e9f213b54d78d67bc69f8 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206376.xml -- Model is restarting -- Adjusting last_iter, 1512, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml last_cpu_time: 5694 upload_file_number: 23 last_iter: 1511 last_upload: 5443200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0732_2020050100_123_989_12206376_1_r187916441 The child process has been launched with process id: 1348389 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1348389:1348389] [20230122:094232:1674373352:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0732 (argv4) batchid: 989 (argv5) wuid: 12206376 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206376 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_734e4ce79bcf9d75a687480617c6b9bd to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip ic_ancil_file: ic_ancil_12206376 ifsdata_file: ifsdata_12206376 climate_data_file: clim_data_12206376 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_cf69bd1ab6ca2467bfd1a3f83c2e0dc0 to: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c49648ee49e9f213b54d78d67bc69f8 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206376.xml -- Model is restarting -- Adjusting last_iter, 1512, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml last_cpu_time: 5701 upload_file_number: 23 last_iter: 1511 last_upload: 5443200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0732_2020050100_123_989_12206376_1_r187916441 The child process has been launched with process id: 1349315 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1349315:1349315] [20230122:094352:1674373432:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0732 (argv4) batchid: 989 (argv5) wuid: 12206376 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206376 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_734e4ce79bcf9d75a687480617c6b9bd to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip ic_ancil_file: ic_ancil_12206376 ifsdata_file: ifsdata_12206376 climate_data_file: clim_data_12206376 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_cf69bd1ab6ca2467bfd1a3f83c2e0dc0 to: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c49648ee49e9f213b54d78d67bc69f8 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206376.xml -- Model is restarting -- Adjusting last_iter, 1512, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml last_cpu_time: 5708 upload_file_number: 23 last_iter: 1511 last_upload: 5443200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0732_2020050100_123_989_12206376_1_r187916441 The child process has been launched with process id: 1352346 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1352346:1352346] [20230122:094514:1674373514:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0732 (argv4) batchid: 989 (argv5) wuid: 12206376 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206376 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_734e4ce79bcf9d75a687480617c6b9bd to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip ic_ancil_file: ic_ancil_12206376 ifsdata_file: ifsdata_12206376 climate_data_file: clim_data_12206376 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_cf69bd1ab6ca2467bfd1a3f83c2e0dc0 to: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c49648ee49e9f213b54d78d67bc69f8 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml last_cpu_time: 0 upload_file_number: 0 last_iter: 0 last_upload: 0 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0732_2020050100_123_989_12206376_1_r187916441 The child process has been launched with process id: 1353126 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1353126:1353126] [20230122:094624:1674373584:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2020050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0732 (argv4) batchid: 989 (argv5) wuid: 12206376 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206376 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_734e4ce79bcf9d75a687480617c6b9bd to: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_ps_0732_2020050100_123_989_12206376.zip ic_ancil_file: ic_ancil_12206376 ifsdata_file: ifsdata_12206376 climate_data_file: clim_data_12206376 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_cf69bd1ab6ca2467bfd1a3f83c2e0dc0 to: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206376.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9c49648ee49e9f213b54d78d67bc69f8 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206376.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206376.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206376.xml -- Model is restarting -- Adjusting last_iter, 1512, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206376.xml last_cpu_time: 7 upload_file_number: 0 last_iter: 1511 last_upload: 0 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0732_2020050100_123_989_12206376_1_r187916441 The child process has been launched with process id: 1408977 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:ardbeg:1:1:1408977:1408977] [20230122:105025:1674377425:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:43:18 STEP 1512 H=1512:00 +CPU= 18.274 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:43:46 STEP 1513 H=1513:00 +CPU= 10.033 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:44:16 STEP 1514 H=1514:00 +CPU= 10.742 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:44:46 STEP 1515 H=1515:00 +CPU= 13.900 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:44:58 STEP 1516 H=1516:00 +CPU= 8.418 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:45:17 STEP 1517 H=1517:00 +CPU= 8.441 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:45:43 STEP 1518 H=1518:00 +CPU= 12.936 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:45:59 STEP 1519 H=1519:00 +CPU= 8.057 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:46:17 STEP 1520 H=1520:00 +CPU= 8.751 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:46:43 STEP 1521 H=1521:00 +CPU= 14.299 Suspend request received from the BOINC client, suspending the child process Abort request received from the BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:35:02 | 1538188 | 22293642 | oifs_43r3_ps_0732_2020050100_123_989_12206376_1 | 5,184,000 | 4,818 | 0.0009 |
09 May 2023 16:23:23 | 1538188 | 22293642 | oifs_43r3_ps_0732_2020050100_123_989_12206376_1 | 2,592,000 | 12,126 | 0.0047 |
09 May 2023 16:23:18 | 1538188 | 22293642 | oifs_43r3_ps_0732_2020050100_123_989_12206376_1 | 1,728,000 | 8,055 | 0.0047 |
09 May 2023 16:23:13 | 1538188 | 22293642 | oifs_43r3_ps_0732_2020050100_123_989_12206376_1 | 864,000 | 4,134 | 0.0048 |
©2024 cpdn.org