Name | oifs_43r3_bl_a01h_2016092300_15_991_12206840_0 |
Workunit | 12206840 |
Created | 13 Feb 2023, 12:06:42 UTC |
Sent | 13 Feb 2023, 12:08:31 UTC |
Report deadline | 14 Apr 2023, 12:08:31 UTC |
Received | 14 Feb 2023, 20:26:02 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Aborted by user |
Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
Computer ID | 1526041 |
Run time | 15 hours 3 min 32 sec |
CPU time | 15 hours 3 min 32 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 2.26 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.11 x86_64-pc-linux-gnu |
Peak working set size | 5,068.81 MB |
Peak swap size | 5,930.78 MB |
Peak disk usage | 1,343.14 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> aborted by user</message> <stderr_txt> ction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01h (argv4) batchid: 991 (argv5) wuid: 12206840 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/22 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip ic_ancil_file: ic_ancil_12206840 ifsdata_file: ifsdata_12206840 climate_data_file: clim_data_12206840 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml -- Model is restarting -- Adjusting last_iter, 742, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml last_cpu_time: 43585 upload_file_number: 7 last_iter: 721 last_upload: 604800 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541 The child process has been launched with process id: 3879 Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:142212:1676384532:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 14:24:37 STEP 720 H= 180:00 +CPU= 54.691 14:25:27 STEP 721 H= 180:15 +CPU= 48.232 14:26:17 STEP 722 H= 180:30 +CPU= 49.043 14:27:09 STEP 723 H= 180:45 +CPU= 50.525 14:27:58 STEP 724 H= 181:00 +CPU= 47.083 14:28:46 STEP 725 H= 181:15 +CPU= 48.174 14:29:37 STEP 726 H= 181:30 +CPU= 49.556 14:30:23 STEP 727 H= 181:45 +CPU= 45.380 14:31:12 STEP 728 H= 182:00 +CPU= 47.207 14:31:55 STEP 729 H= 182:15 +CPU= 42.847 14:32:44 STEP 730 H= 182:30 +CPU= 48.972 14:33:30 STEP 731 H= 182:45 +CPU= 45.570 14:34:23 STEP 732 H= 183:00 +CPU= 53.079 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000732 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000732 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000732 14:35:08 STEP 733 H= 183:15 +CPU= 45.191 14:36:02 STEP 734 H= 183:30 +CPU= 47.440 14:36:53 STEP 735 H= 183:45 +CPU= 49.927 14:37:48 STEP 736 H= 184:00 +CPU= 55.340 14:38:39 STEP 737 H= 184:15 +CPU= 50.405 14:39:30 STEP 738 H= 184:30 +CPU= 51.123 14:40:22 STEP 739 H= 184:45 +CPU= 52.067 14:41:15 STEP 740 H= 185:00 +CPU= 52.280 14:42:05 STEP 741 H= 185:15 +CPU= 49.599 14:42:55 STEP 742 H= 185:30 +CPU= 50.552 14:43:47 STEP 743 H= 185:45 +CPU= 51.192 14:44:52 STEP 744 H= 186:00 +CPU= 65.127 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000744 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000744 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000744 14:45:45 STEP 745 H= 186:15 +CPU= 53.247 14:46:39 STEP 746 H= 186:30 +CPU= 53.608 14:47:31 STEP 747 H= 186:45 +CPU= 51.499 14:48:23 STEP 748 H= 187:00 +CPU= 52.603 14:49:16 STEP 749 H= 187:15 +CPU= 52.053 14:50:07 STEP 750 H= 187:30 +CPU= 51.615 14:50:59 STEP 751 H= 187:45 +CPU= 51.446 14:51:51 STEP 752 H= 188:00 +CPU= 51.915 14:52:46 STEP 753 H= 188:15 +CPU= 54.543 14:53:40 STEP 754 H= 188:30 +CPU= 54.470 14:54:32 STEP 755 H= 188:45 +CPU= 51.672 14:55:38 STEP 756 H= 189:00 +CPU= 65.571 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000756 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000756 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000756 14:56:31 STEP 757 H= 189:15 +CPU= 53.064 14:57:22 STEP 758 H= 189:30 +CPU= 50.601 14:58:14 STEP 759 H= 189:45 +CPU= 51.505 14:59:05 STEP 760 H= 190:00 +CPU= 51.212 14:59:54 STEP 761 H= 190:15 +CPU= 49.270 15:00:45 STEP 762 H= 190:30 +CPU= 50.887 15:01:36 STEP 763 H= 190:45 +CPU= 50.190 15:02:32 STEP 764 H= 191:00 +CPU= 56.039 15:03:23 STEP 765 H= 191:15 +CPU= 50.439 15:04:14 STEP 766 H= 191:30 +CPU= 50.635 15:05:04 STEP 767 H= 191:45 +CPU= 50.809 15:06:23 STEP 768 H= 192:00 +CPU= 71.712 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000768 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000768 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000768 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000732 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000744 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMGGh7zg+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMSHh7zg+000756 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840/ICMUAh7zg+000756 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541_7.zip 15:07:17 STEP 769 H= 192:15 +CPU= 50.607 Uploading the intermediate file: upload_file_7.zip 15:08:09 STEP 770 H= 192:30 +CPU= 50.545 15:09:00 STEP 771 H= 192:45 +CPU= 51.547 15:09:53 STEP 772 H= 193:00 +CPU= 52.561 15:10:42 STEP 773 H= 193:15 +CPU= 49.381 15:11:38 STEP 774 H= 193:30 +CPU= 54.094 15:12:30 STEP 775 H= 193:45 +CPU= 52.097 15:13:20 STEP 776 H= 194:00 +CPU= 49.607 15:14:12 STEP 777 H= 194:15 +CPU= 51.391 15:15:02 STEP 778 H= 194:30 +CPU= 50.281 15:15:55 STEP 779 H= 194:45 +CPU= 52.947 15:17:02 STEP 780 H= 195:00 +CPU= 67.213 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000780 15:17:55 STEP 781 H= 195:15 +CPU= 51.890 15:18:49 STEP 782 H= 195:30 +CPU= 54.083 15:19:44 STEP 783 H= 195:45 +CPU= 55.252 15:20:40 STEP 784 H= 196:00 +CPU= 54.899 15:21:35 STEP 785 H= 196:15 +CPU= 54.514 15:22:27 STEP 786 H= 196:30 +CPU= 51.944 15:23:22 STEP 787 H= 196:45 +CPU= 55.120 15:24:20 STEP 788 H= 197:00 +CPU= 57.870 15:25:14 STEP 789 H= 197:15 +CPU= 53.750 15:26:06 STEP 790 H= 197:30 +CPU= 51.339 15:26:57 STEP 791 H= 197:45 +CPU= 51.357 15:28:07 STEP 792 H= 198:00 +CPU= 69.718 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000792 15:28:57 STEP 793 H= 198:15 +CPU= 49.672 15:29:48 STEP 794 H= 198:30 +CPU= 50.559 15:30:38 STEP 795 H= 198:45 +CPU= 50.257 15:31:30 STEP 796 H= 199:00 +CPU= 51.820 15:32:18 STEP 797 H= 199:15 +CPU= 47.707 15:33:10 STEP 798 H= 199:30 +CPU= 52.104 15:33:59 STEP 799 H= 199:45 +CPU= 49.210 15:34:54 STEP 800 H= 200:00 +CPU= 54.822 15:35:48 STEP 801 H= 200:15 +CPU= 53.750 15:36:38 STEP 802 H= 200:30 +CPU= 49.752 15:37:27 STEP 803 H= 200:45 +CPU= 48.790 15:38:30 STEP 804 H= 201:00 +CPU= 62.860 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000804 15:39:19 STEP 805 H= 201:15 +CPU= 48.831 15:40:07 STEP 806 H= 201:30 +CPU= 47.819 15:40:55 STEP 807 H= 201:45 +CPU= 47.845 15:41:45 STEP 808 H= 202:00 +CPU= 49.639 15:42:34 STEP 809 H= 202:15 +CPU= 48.647 15:43:22 STEP 810 H= 202:30 +CPU= 48.632 15:44:11 STEP 811 H= 202:45 +CPU= 48.475 15:45:01 STEP 812 H= 203:00 +CPU= 49.866 15:45:51 STEP 813 H= 203:15 +CPU= 49.756 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5397MB (heap), 4884MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 5397 MB (maxheap), 4884 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LAPINEA [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARCINA [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LASCAW [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154604:1676389564:5031.792] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LASCAW_CLA [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154605:1676389565:5032.793] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:154605:1676389565:5032.793] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 000000000081FCF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8BB20 Unknown Unknown Unknown oifs_43r3_model.e 0000000000FCA684 lascaw_cla_ 198 lascaw_cla.F90 oifs_43r3_model.e 0000000000FB552E lascaw_ 866 lascaw.F90 oifs_43r3_model.e 0000000001064965 larcina_ 319 larcina.F90 oifs_43r3_model.e 0000000001059F2D lapinea_ 325 lapinea.F90 oifs_43r3_model.e 0000000000FF4E6C call_sl_ 300 call_sl.F90 oifs_43r3_model.e 0000000001081A7D call_sl_heap_ 94 call_sl_heap.F90 oifs_43r3_model.e 0000000000E67648 gp_model_ 533 gp_model.F90 oifs_43r3_model.e 00000000015C1E62 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 00000000015AC40E scan2m_ 535 scan2m.F90 oifs_43r3_model.e 00000000006620B4 stepo_ 327 stepo.F90 oifs_43r3_model.e 00000000004A0E7C cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 000000000049DC29 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 000000000049CE72 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 000000000049CD68 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040276F MAIN__ 96 master.F90 oifs_43r3_model.e 000000000040270D Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8287A Unknown Unknown Unknown oifs_43r3_model.e 0000000001D840D7 Unknown Unknown Unknown oifs_43r3_model.e 00000000004025E5 Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01h (argv4) batchid: 991 (argv5) wuid: 12206840 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/22 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip ic_ancil_file: ic_ancil_12206840 ifsdata_file: ifsdata_12206840 climate_data_file: clim_data_12206840 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml -- Model is restarting -- Adjusting last_iter, 814, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml last_cpu_time: 48487 upload_file_number: 8 last_iter: 769 last_upload: 691200 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541 The child process has been launched with process id: 3859 Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3859:3859] [20230214:155101:1676389861:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:52:21 STEP 768 H= 192:00 +CPU= 47.831 15:53:12 STEP 769 H= 192:15 +CPU= 49.824 15:54:03 STEP 770 H= 192:30 +CPU= 50.418 15:54:47 STEP 771 H= 192:45 +CPU= 44.067 15:55:46 STEP 772 H= 193:00 +CPU= 55.935 15:56:36 STEP 773 H= 193:15 +CPU= 48.172 15:57:18 STEP 774 H= 193:30 +CPU= 42.145 15:58:04 STEP 775 H= 193:45 +CPU= 45.166 15:58:49 STEP 776 H= 194:00 +CPU= 45.112 15:59:36 STEP 777 H= 194:15 +CPU= 46.659 16:00:26 STEP 778 H= 194:30 +CPU= 49.661 16:01:14 STEP 779 H= 194:45 +CPU= 47.503 16:02:15 STEP 780 H= 195:00 +CPU= 60.574 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000780 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000780 16:03:05 STEP 781 H= 195:15 +CPU= 49.724 16:03:53 STEP 782 H= 195:30 +CPU= 47.976 16:04:42 STEP 783 H= 195:45 +CPU= 48.750 16:05:35 STEP 784 H= 196:00 +CPU= 53.222 16:06:24 STEP 785 H= 196:15 +CPU= 48.762 16:07:09 STEP 786 H= 196:30 +CPU= 44.422 16:07:57 STEP 787 H= 196:45 +CPU= 47.349 16:08:52 STEP 788 H= 197:00 +CPU= 53.347 16:09:43 STEP 789 H= 197:15 +CPU= 50.819 16:10:35 STEP 790 H= 197:30 +CPU= 50.962 16:11:40 STEP 791 H= 197:45 +CPU= 52.837 16:12:46 STEP 792 H= 198:00 +CPU= 63.344 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000792 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000792 16:13:36 STEP 793 H= 198:15 +CPU= 49.405 16:14:26 STEP 794 H= 198:30 +CPU= 49.123 16:15:15 STEP 795 H= 198:45 +CPU= 48.040 16:16:08 STEP 796 H= 199:00 +CPU= 52.037 16:17:00 STEP 797 H= 199:15 +CPU= 51.738 16:17:52 STEP 798 H= 199:30 +CPU= 49.076 16:18:50 STEP 799 H= 199:45 +CPU= 49.637 16:19:45 STEP 800 H= 200:00 +CPU= 51.294 16:20:42 STEP 801 H= 200:15 +CPU= 52.111 16:21:34 STEP 802 H= 200:30 +CPU= 49.710 16:22:25 STEP 803 H= 200:45 +CPU= 48.678 16:23:31 STEP 804 H= 201:00 +CPU= 63.472 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000804 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000804 16:24:24 STEP 805 H= 201:15 +CPU= 52.132 16:25:16 STEP 806 H= 201:30 +CPU= 50.512 16:26:14 STEP 807 H= 201:45 +CPU= 51.934 16:27:09 STEP 808 H= 202:00 +CPU= 51.388 16:28:07 STEP 809 H= 202:15 +CPU= 52.669 16:29:00 STEP 810 H= 202:30 +CPU= 51.227 16:29:54 STEP 811 H= 202:45 +CPU= 50.228 16:30:54 STEP 812 H= 203:00 +CPU= 54.800 16:31:51 STEP 813 H= 203:15 +CPU= 53.261 16:32:45 STEP 814 H= 203:30 +CPU= 51.980 16:33:37 STEP 815 H= 203:45 +CPU= 49.226 16:34:51 STEP 816 H= 204:00 +CPU= 69.507 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000816 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000816 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000816 16:35:43 STEP 817 H= 204:15 +CPU= 50.316 16:36:40 STEP 818 H= 204:30 +CPU= 51.031 16:37:31 STEP 819 H= 204:45 +CPU= 48.998 16:38:30 STEP 820 H= 205:00 +CPU= 54.346 16:39:23 STEP 821 H= 205:15 +CPU= 50.249 16:40:18 STEP 822 H= 205:30 +CPU= 51.627 16:41:14 STEP 823 H= 205:45 +CPU= 51.342 16:42:09 STEP 824 H= 206:00 +CPU= 51.337 16:43:03 STEP 825 H= 206:15 +CPU= 51.127 16:43:57 STEP 826 H= 206:30 +CPU= 51.544 16:44:46 STEP 827 H= 206:45 +CPU= 48.548 16:45:54 STEP 828 H= 207:00 +CPU= 65.021 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000828 16:46:44 STEP 829 H= 207:15 +CPU= 49.700 16:47:31 STEP 830 H= 207:30 +CPU= 46.911 16:48:20 STEP 831 H= 207:45 +CPU= 49.358 16:49:13 STEP 832 H= 208:00 +CPU= 51.665 16:50:01 STEP 833 H= 208:15 +CPU= 47.759 16:50:54 STEP 834 H= 208:30 +CPU= 51.983 16:51:42 STEP 835 H= 208:45 +CPU= 47.285 16:52:32 STEP 836 H= 209:00 +CPU= 50.321 16:53:22 STEP 837 H= 209:15 +CPU= 49.048 16:54:10 STEP 838 H= 209:30 +CPU= 47.701 16:55:01 STEP 839 H= 209:45 +CPU= 50.927 16:56:07 STEP 840 H= 210:00 +CPU= 64.470 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000840 16:57:01 STEP 841 H= 210:15 +CPU= 52.673 16:57:53 STEP 842 H= 210:30 +CPU= 50.754 16:58:46 STEP 843 H= 210:45 +CPU= 52.049 16:59:40 STEP 844 H= 211:00 +CPU= 52.488 17:00:34 STEP 845 H= 211:15 +CPU= 51.590 17:01:20 STEP 846 H= 211:30 +CPU= 46.431 17:02:05 STEP 847 H= 211:45 +CPU= 44.228 17:02:56 STEP 848 H= 212:00 +CPU= 51.058 17:03:46 STEP 849 H= 212:15 +CPU= 49.426 (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01h (argv4) batchid: 991 (argv5) wuid: 12206840 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/22 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip ic_ancil_file: ic_ancil_12206840 ifsdata_file: ifsdata_12206840 climate_data_file: clim_data_12206840 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml -- Model is restarting -- Adjusting last_iter, 850, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml last_cpu_time: 52719 upload_file_number: 8 last_iter: 817 last_upload: 691200 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541 The child process has been launched with process id: 3860 Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3860:3860] [20230214:185204:1676400724:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 18:54:21 STEP 816 H= 204:00 +CPU= 55.939 18:55:11 STEP 817 H= 204:15 +CPU= 49.651 18:56:05 STEP 818 H= 204:30 +CPU= 52.957 18:57:01 STEP 819 H= 204:45 +CPU= 53.697 18:57:52 STEP 820 H= 205:00 +CPU= 50.425 18:58:42 STEP 821 H= 205:15 +CPU= 49.553 18:59:32 STEP 822 H= 205:30 +CPU= 49.522 19:00:20 STEP 823 H= 205:45 +CPU= 48.411 19:01:08 STEP 824 H= 206:00 +CPU= 47.381 19:01:55 STEP 825 H= 206:15 +CPU= 46.405 19:02:38 STEP 826 H= 206:30 +CPU= 42.856 19:03:22 STEP 827 H= 206:45 +CPU= 43.628 19:04:25 STEP 828 H= 207:00 +CPU= 63.272 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000828 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000828 19:05:08 STEP 829 H= 207:15 +CPU= 42.297 19:05:51 STEP 830 H= 207:30 +CPU= 42.911 19:06:35 STEP 831 H= 207:45 +CPU= 43.860 19:07:24 STEP 832 H= 208:00 +CPU= 48.931 19:08:12 STEP 833 H= 208:15 +CPU= 47.713 19:08:59 STEP 834 H= 208:30 +CPU= 46.308 19:09:51 STEP 835 H= 208:45 +CPU= 51.961 19:10:47 STEP 836 H= 209:00 +CPU= 53.749 19:11:45 STEP 837 H= 209:15 +CPU= 54.276 19:12:42 STEP 838 H= 209:30 +CPU= 53.887 19:13:36 STEP 839 H= 209:45 +CPU= 52.705 19:14:46 STEP 840 H= 210:00 +CPU= 66.733 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMGGh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMSHh7zg+000840 Moving to projects directory: /var/lib/boinc-client/slots/22/ICMUAh7zg+000840 19:15:39 STEP 841 H= 210:15 +CPU= 51.820 (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a01h (argv4) batchid: 991 (argv5) wuid: 12206840 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/22 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12206840 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_d887486f6a3aabe4c55374c1725be1de to: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/22/oifs_43r3_bl_a01h_2016092300_15_991_12206840.zip ic_ancil_file: ic_ancil_12206840 ifsdata_file: ifsdata_12206840 climate_data_file: clim_data_12206840 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/22/ic_ancil_12206840.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_155e44070232d2cfdcf7069aabe5bcf7 to: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/22/ifsdata/ifsdata_12206840.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/22/159l_2/clim_data_12206840.zip Checking for progress XML file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml Opened progress file ok : /var/lib/boinc-client/slots/22/progress_file_12206840.xml -- Model is restarting -- Adjusting last_iter, 842, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/22/progress_file_12206840.xml last_cpu_time: 54087 upload_file_number: 8 last_iter: 817 last_upload: 691200 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a01h_2016092300_15_991_12206840_0_r968148541 The child process has been launched with process id: 3879 Executing the command: /var/lib/boinc-client/slots/22/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:chonky:1:1:3879:3879] [20230214:192219:1676402539:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:24:33 STEP 816 H= 204:00 +CPU= 52.374 19:25:16 STEP 817 H= 204:15 +CPU= 43.298 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org