Name | oifs_43r3_ps_0178_1990050100_123_959_12175822_1 |
Workunit | 12175822 |
Created | 4 Feb 2023, 18:30:50 UTC |
Sent | 4 Feb 2023, 18:32:20 UTC |
Report deadline | 5 Apr 2023, 18:32:20 UTC |
Received | 6 Feb 2023, 20:31:01 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1499026 |
Run time | 3 hours 33 min 35 sec |
CPU time | 3 hours 31 min 52 sec |
Validate state | Invalid |
Credit | 2,419.20 |
Device peak FLOPS | 6.61 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,767.12 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 7,682.19 MB |
Stderr | <core_client_version>7.14.2</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> it/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7201:7201] [20230206:205041:1675713041:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:51:12 STEP 912 H= 912:00 +CPU= 16.861 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:51:30 STEP 913 H= 913:00 +CPU= 9.319 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:51:45 STEP 914 H= 914:00 +CPU= 9.453 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:52:10 STEP 915 H= 915:00 +CPU= 14.708 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:52:26 STEP 916 H= 916:00 +CPU= 9.934 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:52:44 STEP 917 H= 917:00 +CPU= 9.778 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:53:09 STEP 918 H= 918:00 +CPU= 14.924 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:53:26 STEP 919 H= 919:00 +CPU= 9.695 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:53:42 STEP 920 H= 920:00 +CPU= 9.954 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:54:07 STEP 921 H= 921:00 +CPU= 15.060 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:54:23 STEP 922 H= 922:00 +CPU= 9.490 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:54:38 STEP 923 H= 923:00 +CPU= 9.146 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:55:05 STEP 924 H= 924:00 +CPU= 16.435 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000924 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1990050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0178 (argv4) batchid: 959 (argv5) wuid: 12175822 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_f79d42921fccf0618f1692b12ca1ce67 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0178_1990050100_123_959_12175822.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0178_1990050100_123_959_12175822.zip ic_ancil_file: ic_ancil_12175822 ifsdata_file: ifsdata_12175822 climate_data_file: clim_data_12175822 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_6d9de39d312bfb1658ad471bdd45d7e8 to: /var/lib/boinc-client/slots/1/ic_ancil_12175822.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12175822.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_17b35a9fcb1f51006ad85a9bb1122059 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12175822.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12175822.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12175822.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12175822.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12175822.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12175822.xml -- Model is restarting -- Adjusting last_iter, 925, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12175822.xml last_cpu_time: 12184 upload_file_number: 38 last_iter: 911 last_upload: 3283200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0178_1990050100_123_959_12175822_1_r1137890728 The child process has been launched with process id: 7368 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7368:7368] [20230206:205622:1675713382:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:56:55 STEP 912 H= 912:00 +CPU= 16.447 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:57:14 STEP 913 H= 913:00 +CPU= 9.334 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:57:30 STEP 914 H= 914:00 +CPU= 10.129 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:57:55 STEP 915 H= 915:00 +CPU= 15.033 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:58:11 STEP 916 H= 916:00 +CPU= 9.762 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:58:27 STEP 917 H= 917:00 +CPU= 9.375 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:58:52 STEP 918 H= 918:00 +CPU= 15.156 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:59:11 STEP 919 H= 919:00 +CPU= 9.532 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:59:27 STEP 920 H= 920:00 +CPU= 9.493 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 20:59:49 STEP 921 H= 921:00 +CPU= 14.505 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:00:07 STEP 922 H= 922:00 +CPU= 9.547 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:00:23 STEP 923 H= 923:00 +CPU= 9.787 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:00:51 STEP 924 H= 924:00 +CPU= 16.554 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000924 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000924 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:01:09 STEP 925 H= 925:00 +CPU= 9.292 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:01:22 STEP 926 H= 926:00 +CPU= 9.943 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:01:50 STEP 927 H= 927:00 +CPU= 15.314 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:02:06 STEP 928 H= 928:00 +CPU= 10.120 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:02:21 STEP 929 H= 929:00 +CPU= 9.573 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:02:46 STEP 930 H= 930:00 +CPU= 14.981 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:03:02 STEP 931 H= 931:00 +CPU= 9.624 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:03:18 STEP 932 H= 932:00 +CPU= 9.961 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:03:43 STEP 933 H= 933:00 +CPU= 15.149 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:04:01 STEP 934 H= 934:00 +CPU= 10.261 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:04:19 STEP 935 H= 935:00 +CPU= 9.927 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:04:48 STEP 936 H= 936:00 +CPU= 16.972 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:05:04 STEP 937 H= 937:00 +CPU= 9.442 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000936 Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000936 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822/ICMGGhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822/ICMSHhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822/ICMUAhq0f+000912 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822/ICMGGhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822/ICMSHhq0f+000924 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822/ICMUAhq0f+000924 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0178_1990050100_123_959_12175822_1_r1137890728_38.zip Uploading the intermediate file: upload_file_38.zip 21:05:14 STEP 938 H= 938:00 +CPU= 9.612 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:05:32 STEP 939 H= 939:00 +CPU= 14.644 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:05:48 STEP 940 H= 940:00 +CPU= 9.531 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:06:05 STEP 941 H= 941:00 +CPU= 9.563 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:06:33 STEP 942 H= 942:00 +CPU= 15.194 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:06:48 STEP 943 H= 943:00 +CPU= 9.502 Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1990050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0178 (argv4) batchid: 959 (argv5) wuid: 12175822 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_f79d42921fccf0618f1692b12ca1ce67 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0178_1990050100_123_959_12175822.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0178_1990050100_123_959_12175822.zip ic_ancil_file: ic_ancil_12175822 ifsdata_file: ifsdata_12175822 climate_data_file: clim_data_12175822 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_6d9de39d312bfb1658ad471bdd45d7e8 to: /var/lib/boinc-client/slots/1/ic_ancil_12175822.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12175822.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_17b35a9fcb1f51006ad85a9bb1122059 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12175822.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12175822.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12175822.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12175822.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12175822.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12175822.xml -- Model is restarting -- Adjusting last_iter, 943, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12175822.xml last_cpu_time: 12550 upload_file_number: 39 last_iter: 935 last_upload: 3369600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0178_1990050100_123_959_12175822_1_r1137890728 The child process has been launched with process id: 7708 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7708:7708] [20230206:210755:1675714075:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:08:22 STEP 936 H= 936:00 +CPU= 16.379 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:08:39 STEP 937 H= 937:00 +CPU= 9.499 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:08:57 STEP 938 H= 938:00 +CPU= 9.788 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:09:22 STEP 939 H= 939:00 +CPU= 14.539 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:09:37 STEP 940 H= 940:00 +CPU= 9.599 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:09:53 STEP 941 H= 941:00 +CPU= 9.260 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:10:18 STEP 942 H= 942:00 +CPU= 15.575 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:10:36 STEP 943 H= 943:00 +CPU= 9.324 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:10:51 STEP 944 H= 944:00 +CPU= 9.393 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:11:16 STEP 945 H= 945:00 +CPU= 14.550 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:11:31 STEP 946 H= 946:00 +CPU= 9.491 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:11:47 STEP 947 H= 947:00 +CPU= 9.446 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1990050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0178 (argv4) batchid: 959 (argv5) wuid: 12175822 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/1 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12175822 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_f79d42921fccf0618f1692b12ca1ce67 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0178_1990050100_123_959_12175822.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0178_1990050100_123_959_12175822.zip ic_ancil_file: ic_ancil_12175822 ifsdata_file: ifsdata_12175822 climate_data_file: clim_data_12175822 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_6d9de39d312bfb1658ad471bdd45d7e8 to: /var/lib/boinc-client/slots/1/ic_ancil_12175822.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12175822.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_17b35a9fcb1f51006ad85a9bb1122059 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12175822.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12175822.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12175822.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12175822.zip Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12175822.xml Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12175822.xml -- Model is restarting -- Adjusting last_iter, 948, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12175822.xml last_cpu_time: 12689 upload_file_number: 39 last_iter: 935 last_upload: 3369600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0178_1990050100_123_959_12175822_1_r1137890728 The child process has been launched with process id: 7844 Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:DESKTOP-6SJQPI6:1:1:7844:7844] [20230206:211256:1675714376:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 21:13:22 STEP 936 H= 936:00 +CPU= 15.170 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:47:53 | 1499026 | 22305601 | oifs_43r3_ps_0178_1990050100_123_959_12175822_1 | 1,728,000 | 5,531 | 0.0032 |
09 May 2023 16:47:52 | 1499026 | 22305601 | oifs_43r3_ps_0178_1990050100_123_959_12175822_1 | 864,000 | 2,814 | 0.0033 |
©2024 cpdn.org