Name | oifs_43r3_bl_a01b_2016092300_15_991_12206834_0 |
Workunit | 12206834 |
Created | 13 Feb 2023, 12:06:02 UTC |
Sent | 13 Feb 2023, 12:08:31 UTC |
Report deadline | 14 Apr 2023, 12:08:31 UTC |
Received | 14 Feb 2023, 20:26:02 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Aborted by user |
Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
Computer ID | 1526041 |
Run time | 15 hours 9 min 31 sec |
CPU time | 15 hours 9 min 31 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 2.26 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.11 x86_64-pc-linux-gnu |
Peak working set size | 5,072.46 MB |
Peak swap size | 5,930.79 MB |
Peak disk usage | 1,292.08 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> aborted by user</message> <stderr_txt> b (argv4) batchid: 991 (argv5) wuid: 12206834 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip ic_ancil_file: ic_ancil_12206834 ifsdata_file: ifsdata_12206834 climate_data_file: clim_data_12206834 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml -- Model is restarting -- Adjusting last_iter, 741, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml last_cpu_time: 43889 upload_file_number: 7 last_iter: 721 last_upload: 605700 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939 The child process has been launched with process id: 3880 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:142212:1676384532:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 14:24:40 STEP 720 H= 180:00 +CPU= 57.860 14:25:28 STEP 721 H= 180:15 +CPU= 48.401 14:26:19 STEP 722 H= 180:30 +CPU= 50.418 14:27:11 STEP 723 H= 180:45 +CPU= 50.651 14:28:03 STEP 724 H= 181:00 +CPU= 52.147 14:28:58 STEP 725 H= 181:15 +CPU= 53.709 14:29:52 STEP 726 H= 181:30 +CPU= 53.535 14:30:43 STEP 727 H= 181:45 +CPU= 50.732 14:31:35 STEP 728 H= 182:00 +CPU= 51.167 14:32:27 STEP 729 H= 182:15 +CPU= 52.198 14:33:20 STEP 730 H= 182:30 +CPU= 51.953 14:34:09 STEP 731 H= 182:45 +CPU= 48.691 14:35:15 STEP 732 H= 183:00 +CPU= 65.029 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000732 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000732 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000732 14:36:20 STEP 733 H= 183:15 +CPU= 52.015 14:37:13 STEP 734 H= 183:30 +CPU= 52.380 14:38:03 STEP 735 H= 183:45 +CPU= 49.449 14:38:56 STEP 736 H= 184:00 +CPU= 52.846 14:39:45 STEP 737 H= 184:15 +CPU= 49.036 14:40:34 STEP 738 H= 184:30 +CPU= 48.462 14:41:25 STEP 739 H= 184:45 +CPU= 51.063 14:42:17 STEP 740 H= 185:00 +CPU= 52.165 14:43:08 STEP 741 H= 185:15 +CPU= 50.655 14:43:59 STEP 742 H= 185:30 +CPU= 50.405 14:44:50 STEP 743 H= 185:45 +CPU= 51.191 14:45:55 STEP 744 H= 186:00 +CPU= 64.295 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000744 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000744 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000744 14:46:47 STEP 745 H= 186:15 +CPU= 51.954 14:47:40 STEP 746 H= 186:30 +CPU= 52.747 14:48:29 STEP 747 H= 186:45 +CPU= 49.490 14:49:21 STEP 748 H= 187:00 +CPU= 51.933 14:50:14 STEP 749 H= 187:15 +CPU= 52.627 14:51:03 STEP 750 H= 187:30 +CPU= 48.824 14:51:50 STEP 751 H= 187:45 +CPU= 46.422 14:52:41 STEP 752 H= 188:00 +CPU= 50.386 14:53:31 STEP 753 H= 188:15 +CPU= 49.756 14:54:19 STEP 754 H= 188:30 +CPU= 48.247 14:55:07 STEP 755 H= 188:45 +CPU= 47.523 14:56:07 STEP 756 H= 189:00 +CPU= 60.098 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000756 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000756 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000756 14:56:56 STEP 757 H= 189:15 +CPU= 48.516 14:57:46 STEP 758 H= 189:30 +CPU= 50.313 14:58:35 STEP 759 H= 189:45 +CPU= 48.315 14:59:27 STEP 760 H= 190:00 +CPU= 52.002 15:00:15 STEP 761 H= 190:15 +CPU= 48.236 15:01:07 STEP 762 H= 190:30 +CPU= 51.029 15:01:57 STEP 763 H= 190:45 +CPU= 49.573 15:02:49 STEP 764 H= 191:00 +CPU= 51.917 15:03:39 STEP 765 H= 191:15 +CPU= 49.994 15:04:28 STEP 766 H= 191:30 +CPU= 48.933 15:05:18 STEP 767 H= 191:45 +CPU= 50.074 15:06:32 STEP 768 H= 192:00 +CPU= 67.979 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000768 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000768 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000768 15:07:33 STEP 769 H= 192:15 +CPU= 49.172 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMGGh7zg+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMSHh7zg+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834/ICMUAh7zg+000768 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939_7.zip Uploading the intermediate file: upload_file_7.zip 15:08:21 STEP 770 H= 192:30 +CPU= 48.106 15:09:11 STEP 771 H= 192:45 +CPU= 49.705 15:10:00 STEP 772 H= 193:00 +CPU= 48.533 15:10:49 STEP 773 H= 193:15 +CPU= 48.588 15:11:38 STEP 774 H= 193:30 +CPU= 48.261 15:12:29 STEP 775 H= 193:45 +CPU= 50.913 15:13:23 STEP 776 H= 194:00 +CPU= 52.996 15:14:12 STEP 777 H= 194:15 +CPU= 48.958 15:15:02 STEP 778 H= 194:30 +CPU= 50.471 15:15:54 STEP 779 H= 194:45 +CPU= 51.703 15:16:59 STEP 780 H= 195:00 +CPU= 64.786 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000780 15:17:52 STEP 781 H= 195:15 +CPU= 52.131 15:18:44 STEP 782 H= 195:30 +CPU= 52.009 15:19:35 STEP 783 H= 195:45 +CPU= 50.617 15:20:26 STEP 784 H= 196:00 +CPU= 50.616 15:21:14 STEP 785 H= 196:15 +CPU= 47.707 15:22:02 STEP 786 H= 196:30 +CPU= 47.929 15:22:52 STEP 787 H= 196:45 +CPU= 49.522 15:23:43 STEP 788 H= 197:00 +CPU= 51.156 15:24:32 STEP 789 H= 197:15 +CPU= 48.359 15:25:22 STEP 790 H= 197:30 +CPU= 50.512 15:26:12 STEP 791 H= 197:45 +CPU= 49.888 15:27:14 STEP 792 H= 198:00 +CPU= 61.453 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000792 15:28:03 STEP 793 H= 198:15 +CPU= 48.724 15:28:52 STEP 794 H= 198:30 +CPU= 48.740 15:29:41 STEP 795 H= 198:45 +CPU= 48.960 15:30:33 STEP 796 H= 199:00 +CPU= 51.256 15:31:25 STEP 797 H= 199:15 +CPU= 51.840 15:32:16 STEP 798 H= 199:30 +CPU= 51.165 15:33:06 STEP 799 H= 199:45 +CPU= 49.462 15:34:00 STEP 800 H= 200:00 +CPU= 53.700 15:34:48 STEP 801 H= 200:15 +CPU= 48.498 15:35:38 STEP 802 H= 200:30 +CPU= 49.170 15:36:26 STEP 803 H= 200:45 +CPU= 48.632 15:37:31 STEP 804 H= 201:00 +CPU= 64.854 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000804 15:38:21 STEP 805 H= 201:15 +CPU= 48.968 15:39:10 STEP 806 H= 201:30 +CPU= 48.900 15:39:59 STEP 807 H= 201:45 +CPU= 48.952 15:40:50 STEP 808 H= 202:00 +CPU= 51.567 15:41:40 STEP 809 H= 202:15 +CPU= 49.523 15:42:30 STEP 810 H= 202:30 +CPU= 49.944 15:43:19 STEP 811 H= 202:45 +CPU= 49.227 15:44:11 STEP 812 H= 203:00 +CPU= 51.521 15:45:00 STEP 813 H= 203:15 +CPU= 48.656 15:45:52 STEP 814 H= 203:30 +CPU= 51.574 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5397MB (heap), 4884MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil)) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 5397 MB (maxheap), 4884 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.765] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004) [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LAPINEA [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARMES [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARCINA [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154604:1676389564:5031.766] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LASCAW [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154605:1676389565:5032.766] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:chonky:1:1:3880:3880] [20230214:154605:1676389565:5032.766] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 000000000081FCF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8BB20 Unknown Unknown Unknown oifs_43r3_model.e 0000000000FAF537 lascaw_ 470 lascaw.F90 oifs_43r3_model.e 0000000001064965 larcina_ 319 larcina.F90 oifs_43r3_model.e 0000000001062038 larmes_ 603 larmes.F90 oifs_43r3_model.e 0000000001058F5D lapinea_ 270 lapinea.F90 oifs_43r3_model.e 0000000000FF4E6C call_sl_ 300 call_sl.F90 oifs_43r3_model.e 0000000001081A7D call_sl_heap_ 94 call_sl_heap.F90 oifs_43r3_model.e 0000000000E67648 gp_model_ 533 gp_model.F90 oifs_43r3_model.e 00000000015C1E62 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 00000000015AC40E scan2m_ 535 scan2m.F90 oifs_43r3_model.e 00000000006620B4 stepo_ 327 stepo.F90 oifs_43r3_model.e 00000000004A0E7C cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 000000000049DC29 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 000000000049CE72 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 000000000049CD68 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040276F MAIN__ 96 master.F90 oifs_43r3_model.e 000000000040270D Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8287A Unknown Unknown Unknown oifs_43r3_model.e 0000000001D840D7 Unknown Unknown Unknown oifs_43r3_model.e 00000000004025E5 Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01b (argv4) batchid: 991 (argv5) wuid: 12206834 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip ic_ancil_file: ic_ancil_12206834 ifsdata_file: ifsdata_12206834 climate_data_file: clim_data_12206834 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml -- Model is restarting -- Adjusting last_iter, 815, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml last_cpu_time: 48781 upload_file_number: 8 last_iter: 769 last_upload: 692100 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939 The child process has been launched with process id: 3855 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3855:3855] [20230214:155101:1676389861:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:52:27 STEP 768 H= 192:00 +CPU= 51.603 15:53:15 STEP 769 H= 192:15 +CPU= 47.261 15:54:00 STEP 770 H= 192:30 +CPU= 44.260 15:54:44 STEP 771 H= 192:45 +CPU= 43.979 15:55:36 STEP 772 H= 193:00 +CPU= 51.181 15:56:25 STEP 773 H= 193:15 +CPU= 48.685 15:57:10 STEP 774 H= 193:30 +CPU= 45.205 15:57:55 STEP 775 H= 193:45 +CPU= 44.865 15:58:40 STEP 776 H= 194:00 +CPU= 44.533 15:59:27 STEP 777 H= 194:15 +CPU= 46.932 16:00:12 STEP 778 H= 194:30 +CPU= 44.991 16:00:57 STEP 779 H= 194:45 +CPU= 44.333 16:01:55 STEP 780 H= 195:00 +CPU= 57.181 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000780 16:02:42 STEP 781 H= 195:15 +CPU= 46.993 16:03:27 STEP 782 H= 195:30 +CPU= 45.213 16:04:15 STEP 783 H= 195:45 +CPU= 46.946 16:05:04 STEP 784 H= 196:00 +CPU= 49.162 16:05:52 STEP 785 H= 196:15 +CPU= 47.919 16:06:39 STEP 786 H= 196:30 +CPU= 46.966 16:07:30 STEP 787 H= 196:45 +CPU= 50.478 16:08:19 STEP 788 H= 197:00 +CPU= 49.006 16:09:07 STEP 789 H= 197:15 +CPU= 47.568 16:09:56 STEP 790 H= 197:30 +CPU= 48.515 16:10:48 STEP 791 H= 197:45 +CPU= 50.543 16:12:06 STEP 792 H= 198:00 +CPU= 64.866 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000792 16:12:57 STEP 793 H= 198:15 +CPU= 49.075 16:13:46 STEP 794 H= 198:30 +CPU= 48.825 16:14:34 STEP 795 H= 198:45 +CPU= 47.977 16:15:23 STEP 796 H= 199:00 +CPU= 48.605 16:16:14 STEP 797 H= 199:15 +CPU= 49.798 16:17:03 STEP 798 H= 199:30 +CPU= 49.181 16:17:54 STEP 799 H= 199:45 +CPU= 50.293 16:18:50 STEP 800 H= 200:00 +CPU= 54.345 16:19:46 STEP 801 H= 200:15 +CPU= 54.181 16:20:41 STEP 802 H= 200:30 +CPU= 53.494 16:21:34 STEP 803 H= 200:45 +CPU= 51.463 16:22:39 STEP 804 H= 201:00 +CPU= 63.805 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000804 16:23:33 STEP 805 H= 201:15 +CPU= 52.501 16:24:29 STEP 806 H= 201:30 +CPU= 53.532 16:25:22 STEP 807 H= 201:45 +CPU= 51.837 16:26:21 STEP 808 H= 202:00 +CPU= 54.741 16:27:14 STEP 809 H= 202:15 +CPU= 52.433 16:28:13 STEP 810 H= 202:30 +CPU= 55.715 16:29:06 STEP 811 H= 202:45 +CPU= 51.962 16:30:01 STEP 812 H= 203:00 +CPU= 53.561 16:30:54 STEP 813 H= 203:15 +CPU= 52.958 16:31:47 STEP 814 H= 203:30 +CPU= 52.288 16:32:41 STEP 815 H= 203:45 +CPU= 52.263 16:33:55 STEP 816 H= 204:00 +CPU= 71.845 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000816 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000816 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000816 16:34:50 STEP 817 H= 204:15 +CPU= 53.455 16:35:42 STEP 818 H= 204:30 +CPU= 51.806 16:36:35 STEP 819 H= 204:45 +CPU= 52.139 16:37:29 STEP 820 H= 205:00 +CPU= 53.240 16:38:18 STEP 821 H= 205:15 +CPU= 48.406 16:39:08 STEP 822 H= 205:30 +CPU= 50.011 16:40:00 STEP 823 H= 205:45 +CPU= 51.274 16:40:56 STEP 824 H= 206:00 +CPU= 54.986 16:41:50 STEP 825 H= 206:15 +CPU= 52.580 16:42:44 STEP 826 H= 206:30 +CPU= 52.859 16:43:38 STEP 827 H= 206:45 +CPU= 53.136 16:44:46 STEP 828 H= 207:00 +CPU= 67.394 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000828 16:45:39 STEP 829 H= 207:15 +CPU= 52.438 16:46:32 STEP 830 H= 207:30 +CPU= 51.405 16:47:24 STEP 831 H= 207:45 +CPU= 51.403 16:48:16 STEP 832 H= 208:00 +CPU= 52.049 16:49:12 STEP 833 H= 208:15 +CPU= 53.389 16:50:05 STEP 834 H= 208:30 +CPU= 51.905 16:50:59 STEP 835 H= 208:45 +CPU= 52.620 16:51:54 STEP 836 H= 209:00 +CPU= 54.443 16:52:50 STEP 837 H= 209:15 +CPU= 54.509 16:53:45 STEP 838 H= 209:30 +CPU= 54.321 16:54:42 STEP 839 H= 209:45 +CPU= 54.162 16:55:48 STEP 840 H= 210:00 +CPU= 64.833 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000840 16:56:43 STEP 841 H= 210:15 +CPU= 52.900 16:57:37 STEP 842 H= 210:30 +CPU= 52.120 16:58:30 STEP 843 H= 210:45 +CPU= 51.070 16:59:22 STEP 844 H= 211:00 +CPU= 51.545 17:00:19 STEP 845 H= 211:15 +CPU= 54.069 17:01:09 STEP 846 H= 211:30 +CPU= 49.291 17:01:55 STEP 847 H= 211:45 +CPU= 45.626 17:02:44 STEP 848 H= 212:00 +CPU= 47.977 17:03:34 STEP 849 H= 212:15 +CPU= 49.533 (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01b (argv4) batchid: 991 (argv5) wuid: 12206834 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip ic_ancil_file: ic_ancil_12206834 ifsdata_file: ifsdata_12206834 climate_data_file: clim_data_12206834 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml -- Model is restarting -- Adjusting last_iter, 850, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml last_cpu_time: 53076 upload_file_number: 8 last_iter: 817 last_upload: 692100 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939 The child process has been launched with process id: 3861 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3861:3861] [20230214:185204:1676400724:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 18:54:15 STEP 816 H= 204:00 +CPU= 50.660 18:54:59 STEP 817 H= 204:15 +CPU= 43.562 18:55:47 STEP 818 H= 204:30 +CPU= 47.500 18:56:30 STEP 819 H= 204:45 +CPU= 43.197 18:57:13 STEP 820 H= 205:00 +CPU= 42.283 18:57:56 STEP 821 H= 205:15 +CPU= 43.095 18:58:43 STEP 822 H= 205:30 +CPU= 46.214 18:59:29 STEP 823 H= 205:45 +CPU= 45.994 19:00:12 STEP 824 H= 206:00 +CPU= 43.028 19:00:54 STEP 825 H= 206:15 +CPU= 41.224 19:01:44 STEP 826 H= 206:30 +CPU= 49.438 19:02:31 STEP 827 H= 206:45 +CPU= 46.973 19:03:28 STEP 828 H= 207:00 +CPU= 56.398 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000828 19:04:19 STEP 829 H= 207:15 +CPU= 50.240 19:05:02 STEP 830 H= 207:30 +CPU= 42.878 19:05:46 STEP 831 H= 207:45 +CPU= 43.644 19:06:35 STEP 832 H= 208:00 +CPU= 49.354 19:07:24 STEP 833 H= 208:15 +CPU= 48.320 19:08:11 STEP 834 H= 208:30 +CPU= 47.227 19:08:59 STEP 835 H= 208:45 +CPU= 47.335 19:09:53 STEP 836 H= 209:00 +CPU= 50.300 19:10:44 STEP 837 H= 209:15 +CPU= 50.327 19:11:39 STEP 838 H= 209:30 +CPU= 50.593 19:12:31 STEP 839 H= 209:45 +CPU= 51.466 19:13:35 STEP 840 H= 210:00 +CPU= 62.283 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMGGh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMSHh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/23/ICMUAh7zg+000840 19:14:27 STEP 841 H= 210:15 +CPU= 49.395 19:15:17 STEP 842 H= 210:30 +CPU= 48.581 19:16:10 STEP 843 H= 210:45 +CPU= 50.817 (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01b (argv4) batchid: 991 (argv5) wuid: 12206834 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/23 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206834 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_82383272ecd2a6c68ef40d3b5dd703af to: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/23/oifs_43r3_bl_a01b_2016092300_15_991_12206834.zip ic_ancil_file: ic_ancil_12206834 ifsdata_file: ifsdata_12206834 climate_data_file: clim_data_12206834 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/23/ic_ancil_12206834.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f122ec64654a145d9fccb1c0419998b3 to: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/23/ifsdata/ifsdata_12206834.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/23/159l_2/clim_data_12206834.zip Checking for progress XML file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml Opened progress file ok : /var/lib/boinc-client/slots/23/progress_file_12206834.xml -- Model is restarting -- Adjusting last_iter, 844, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/23/progress_file_12206834.xml last_cpu_time: 54446 upload_file_number: 8 last_iter: 817 last_upload: 692100 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01b_2016092300_15_991_12206834_0_r2029125939 The child process has been launched with process id: 3875 Executing the command: /var/lib/boinc-client/slots/23/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3875:3875] [20230214:192219:1676402539:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:24:29 STEP 816 H= 204:00 +CPU= 50.222 19:25:16 STEP 817 H= 204:15 +CPU= 45.570 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org