Name | oifs_43r3_ps_0312_2002050100_123_971_12187956_0 |
Workunit | 12187956 |
Created | 23 Dec 2022, 2:24:29 UTC |
Sent | 30 Dec 2022, 11:23:51 UTC |
Report deadline | 29 Jan 2023, 11:23:51 UTC |
Received | 16 Jan 2023, 12:37:36 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1510976 |
Run time | 8 hours 2 min 17 sec |
CPU time | 1 hours 52 min 31 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 5.70 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,439.29 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,717.68 MB |
Stderr | <core_client_version>7.14.2</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> ] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17921:17921] [20230116:123150:1673868710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17921:17921] [20230116:123150:1673868710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17921:17921] [20230116:123150:1673868710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17921:17921] [20230116:123150:1673868710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17921:17921] [20230116:123150:1673868710:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17921:17921] [20230116:123150:1673868710:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 12:32:30 STEP 312 H= 312:00 +CPU= 34.625 12:32:50 STEP 313 H= 313:00 +CPU= 19.563 12:33:11 STEP 314 H= 314:00 +CPU= 21.094 12:33:44 STEP 315 H= 315:00 +CPU= 32.887 12:34:06 STEP 316 H= 316:00 +CPU= 22.183 12:34:29 STEP 317 H= 317:00 +CPU= 22.745 12:35:01 STEP 318 H= 318:00 +CPU= 32.451 12:35:25 STEP 319 H= 319:00 +CPU= 23.624 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0312 (argv4) batchid: 971 (argv5) wuid: 12187956 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_239d632c8b0d2caba1a1a378dfa44ded to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip ic_ancil_file: ic_ancil_12187956 ifsdata_file: ifsdata_12187956 climate_data_file: clim_data_12187956 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_467ba4db9e321bd857c545c04184a3f3 to: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c8b62f2bd6ffef49f1f9b7b90cbf464 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12187956.xml -- Model is restarting -- Adjusting last_iter, 320, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml last_cpu_time: 4707 upload_file_number: 3 last_iter: 311 last_upload: 1123200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736 The child process has been launched with process id: 17986 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:17986:17986] [20230116:123653:1673869013:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 12:37:38 STEP 312 H= 312:00 +CPU= 39.252 12:38:01 STEP 313 H= 313:00 +CPU= 22.954 12:38:23 STEP 314 H= 314:00 +CPU= 21.239 12:38:57 STEP 315 H= 315:00 +CPU= 33.066 12:39:19 STEP 316 H= 316:00 +CPU= 21.117 12:39:40 STEP 317 H= 317:00 +CPU= 21.010 12:40:13 STEP 318 H= 318:00 +CPU= 32.902 12:40:33 STEP 319 H= 319:00 +CPU= 20.488 12:40:55 STEP 320 H= 320:00 +CPU= 21.619 12:41:29 STEP 321 H= 321:00 +CPU= 33.920 12:41:51 STEP 322 H= 322:00 +CPU= 21.774 12:42:12 STEP 323 H= 323:00 +CPU= 21.349 12:42:49 STEP 324 H= 324:00 +CPU= 36.767 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000324 12:43:11 STEP 325 H= 325:00 +CPU= 22.329 12:43:34 STEP 326 H= 326:00 +CPU= 22.846 12:44:09 STEP 327 H= 327:00 +CPU= 35.146 12:44:31 STEP 328 H= 328:00 +CPU= 22.226 12:44:53 STEP 329 H= 329:00 +CPU= 21.532 12:45:28 STEP 330 H= 330:00 +CPU= 34.639 12:45:50 STEP 331 H= 331:00 +CPU= 22.457 12:46:11 STEP 332 H= 332:00 +CPU= 21.151 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0312 (argv4) batchid: 971 (argv5) wuid: 12187956 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_239d632c8b0d2caba1a1a378dfa44ded to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip ic_ancil_file: ic_ancil_12187956 ifsdata_file: ifsdata_12187956 climate_data_file: clim_data_12187956 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_467ba4db9e321bd857c545c04184a3f3 to: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c8b62f2bd6ffef49f1f9b7b90cbf464 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12187956.xml -- Model is restarting -- Adjusting last_iter, 333, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml last_cpu_time: 5280 upload_file_number: 3 last_iter: 311 last_upload: 1123200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736 The child process has been launched with process id: 18161 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18161:18161] [20230116:124749:1673869669:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 12:48:35 STEP 312 H= 312:00 +CPU= 41.338 12:49:00 STEP 313 H= 313:00 +CPU= 24.439 12:49:23 STEP 314 H= 314:00 +CPU= 23.137 12:49:57 STEP 315 H= 315:00 +CPU= 33.710 12:50:17 STEP 316 H= 316:00 +CPU= 20.312 12:50:40 STEP 317 H= 317:00 +CPU= 23.064 12:51:17 STEP 318 H= 318:00 +CPU= 36.145 12:51:41 STEP 319 H= 319:00 +CPU= 24.234 12:52:05 STEP 320 H= 320:00 +CPU= 24.213 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0312 (argv4) batchid: 971 (argv5) wuid: 12187956 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_239d632c8b0d2caba1a1a378dfa44ded to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip ic_ancil_file: ic_ancil_12187956 ifsdata_file: ifsdata_12187956 climate_data_file: clim_data_12187956 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_467ba4db9e321bd857c545c04184a3f3 to: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c8b62f2bd6ffef49f1f9b7b90cbf464 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12187956.xml -- Model is restarting -- Adjusting last_iter, 321, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml last_cpu_time: 5545 upload_file_number: 3 last_iter: 311 last_upload: 1123200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736 The child process has been launched with process id: 18326 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18326:18326] [20230116:125455:1673870095:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 12:55:41 STEP 312 H= 312:00 +CPU= 40.779 12:56:05 STEP 313 H= 313:00 +CPU= 22.351 12:56:30 STEP 314 H= 314:00 +CPU= 24.474 12:57:01 STEP 315 H= 315:00 +CPU= 31.330 12:57:23 STEP 316 H= 316:00 +CPU= 21.303 12:57:45 STEP 317 H= 317:00 +CPU= 22.048 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0312 (argv4) batchid: 971 (argv5) wuid: 12187956 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_239d632c8b0d2caba1a1a378dfa44ded to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip ic_ancil_file: ic_ancil_12187956 ifsdata_file: ifsdata_12187956 climate_data_file: clim_data_12187956 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_467ba4db9e321bd857c545c04184a3f3 to: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c8b62f2bd6ffef49f1f9b7b90cbf464 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12187956.xml -- Model is restarting -- Adjusting last_iter, 318, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml last_cpu_time: 5708 upload_file_number: 3 last_iter: 311 last_upload: 1123200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736 The child process has been launched with process id: 18355 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18355:18355] [20230116:125856:1673870336:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 12:59:37 STEP 312 H= 312:00 +CPU= 34.652 13:00:00 STEP 313 H= 313:00 +CPU= 23.040 13:00:24 STEP 314 H= 314:00 +CPU= 23.719 13:00:56 STEP 315 H= 315:00 +CPU= 31.871 13:01:20 STEP 316 H= 316:00 +CPU= 24.714 13:01:43 STEP 317 H= 317:00 +CPU= 23.104 13:02:18 STEP 318 H= 318:00 +CPU= 33.971 13:02:38 STEP 319 H= 319:00 +CPU= 20.661 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0312 (argv4) batchid: 971 (argv5) wuid: 12187956 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_239d632c8b0d2caba1a1a378dfa44ded to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip ic_ancil_file: ic_ancil_12187956 ifsdata_file: ifsdata_12187956 climate_data_file: clim_data_12187956 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_467ba4db9e321bd857c545c04184a3f3 to: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c8b62f2bd6ffef49f1f9b7b90cbf464 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12187956.xml -- Model is restarting -- Adjusting last_iter, 320, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml last_cpu_time: 5933 upload_file_number: 3 last_iter: 311 last_upload: 1123200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736 The child process has been launched with process id: 18515 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18515:18515] [20230116:130420:1673870660:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2002050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0312 (argv4) batchid: 971 (argv5) wuid: 12187956 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_239d632c8b0d2caba1a1a378dfa44ded to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0312_2002050100_123_971_12187956.zip ic_ancil_file: ic_ancil_12187956 ifsdata_file: ifsdata_12187956 climate_data_file: clim_data_12187956 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_467ba4db9e321bd857c545c04184a3f3 to: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12187956.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0c8b62f2bd6ffef49f1f9b7b90cbf464 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12187956.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12187956.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12187956.xml -- Model is restarting -- Adjusting last_iter, 312, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12187956.xml last_cpu_time: 5948 upload_file_number: 3 last_iter: 311 last_upload: 1123200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736 The child process has been launched with process id: 18632 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:debby:1:1:18632:18632] [20230116:130738:1673870858:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 13:08:20 STEP 312 H= 312:00 +CPU= 36.356 13:08:46 STEP 313 H= 313:00 +CPU= 23.995 13:09:07 STEP 314 H= 314:00 +CPU= 20.902 13:09:39 STEP 315 H= 315:00 +CPU= 32.012 13:10:04 STEP 316 H= 316:00 +CPU= 22.132 13:10:26 STEP 317 H= 317:00 +CPU= 21.136 13:11:00 STEP 318 H= 318:00 +CPU= 34.275 13:11:23 STEP 319 H= 319:00 +CPU= 23.243 13:11:47 STEP 320 H= 320:00 +CPU= 23.338 13:12:19 STEP 321 H= 321:00 +CPU= 32.018 13:12:41 STEP 322 H= 322:00 +CPU= 21.864 13:13:02 STEP 323 H= 323:00 +CPU= 21.334 13:13:38 STEP 324 H= 324:00 +CPU= 36.176 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000324 13:13:59 STEP 325 H= 325:00 +CPU= 20.507 13:14:22 STEP 326 H= 326:00 +CPU= 22.888 13:14:53 STEP 327 H= 327:00 +CPU= 31.457 13:15:15 STEP 328 H= 328:00 +CPU= 21.360 13:15:37 STEP 329 H= 329:00 +CPU= 22.498 13:16:10 STEP 330 H= 330:00 +CPU= 32.703 13:16:29 STEP 331 H= 331:00 +CPU= 19.304 13:16:50 STEP 332 H= 332:00 +CPU= 20.539 13:17:22 STEP 333 H= 333:00 +CPU= 32.341 13:17:45 STEP 334 H= 334:00 +CPU= 22.677 13:18:07 STEP 335 H= 335:00 +CPU= 22.035 13:18:44 STEP 336 H= 336:00 +CPU= 36.508 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000336 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000336 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000336 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956/ICMGGhq0f+000312 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956/ICMSHhq0f+000312 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956/ICMUAhq0f+000312 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956/ICMGGhq0f+000324 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956/ICMSHhq0f+000324 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187956/ICMUAhq0f+000324 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0312_2002050100_123_971_12187956_0_r905576736_3.zip Uploading the intermediate file: upload_file_3.zip 13:19:06 STEP 337 H= 337:00 +CPU= 21.943 13:19:30 STEP 338 H= 338:00 +CPU= 23.695 13:20:03 STEP 339 H= 339:00 +CPU= 33.180 13:20:25 STEP 340 H= 340:00 +CPU= 21.392 13:20:47 STEP 341 H= 341:00 +CPU= 21.781 Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org