Name | oifs_43r3_bl_a007_2016092300_15_949_12166401_0 |
Workunit | 12166401 |
Created | 14 Dec 2022, 13:58:36 UTC |
Sent | 14 Dec 2022, 13:59:30 UTC |
Report deadline | 13 Jan 2023, 13:59:30 UTC |
Received | 16 Dec 2022, 1:17:41 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1520317 |
Run time | 12 hours 52 min 24 sec |
CPU time | 12 hours 17 min 32 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 8.61 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.07 x86_64-pc-linux-gnu |
Peak working set size | 5,054.83 MB |
Peak swap size | 5,930.81 MB |
Peak disk usage | 6,366.51 MB |
Stderr | <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> +CPU= 39.404 16:03:39 STEP 979 H= 244:45 +CPU= 28.376 16:04:16 STEP 980 H= 245:00 +CPU= 35.232 16:04:52 STEP 981 H= 245:15 +CPU= 33.894 16:05:30 STEP 982 H= 245:30 +CPU= 35.915 16:06:12 STEP 983 H= 245:45 +CPU= 39.645 16:07:01 STEP 984 H= 246:00 +CPU= 46.091 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+000984 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+000984 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+000984 16:07:37 STEP 985 H= 246:15 +CPU= 34.636 16:08:16 STEP 986 H= 246:30 +CPU= 37.137 16:08:48 STEP 987 H= 246:45 +CPU= 31.211 16:09:22 STEP 988 H= 247:00 +CPU= 33.420 16:10:03 STEP 989 H= 247:15 +CPU= 38.711 16:10:43 STEP 990 H= 247:30 +CPU= 37.135 16:11:20 STEP 991 H= 247:45 +CPU= 35.191 16:11:57 STEP 992 H= 248:00 +CPU= 36.194 16:12:32 STEP 993 H= 248:15 +CPU= 33.132 16:13:10 STEP 994 H= 248:30 +CPU= 35.993 16:13:47 STEP 995 H= 248:45 +CPU= 34.740 16:14:39 STEP 996 H= 249:00 +CPU= 50.002 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+000996 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+000996 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+000996 16:15:21 STEP 997 H= 249:15 +CPU= 39.027 16:16:01 STEP 998 H= 249:30 +CPU= 38.119 16:16:41 STEP 999 H= 249:45 +CPU= 38.720 16:17:20 STEP 1000 H= 250:00 +CPU= 38.186 16:18:02 STEP 1001 H= 250:15 +CPU= 39.095 16:18:45 STEP 1002 H= 250:30 +CPU= 40.676 16:19:18 STEP 1003 H= 250:45 +CPU= 32.412 16:20:03 STEP 1004 H= 251:00 +CPU= 41.584 16:20:42 STEP 1005 H= 251:15 +CPU= 36.812 16:21:25 STEP 1006 H= 251:30 +CPU= 41.019 16:21:58 STEP 1007 H= 251:45 +CPU= 31.870 16:23:00 STEP 1008 H= 252:00 +CPU= 59.593 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001008 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001008 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001008 16:23:43 STEP 1009 H= 252:15 +CPU= 41.557 16:24:27 STEP 1010 H= 252:30 +CPU= 41.574 16:25:15 STEP 1011 H= 252:45 +CPU= 45.362 16:25:55 STEP 1012 H= 253:00 +CPU= 39.206 16:26:35 STEP 1013 H= 253:15 +CPU= 39.013 16:27:17 STEP 1014 H= 253:30 +CPU= 39.682 16:28:04 STEP 1015 H= 253:45 +CPU= 45.565 16:28:46 STEP 1016 H= 254:00 +CPU= 39.538 16:29:27 STEP 1017 H= 254:15 +CPU= 38.664 16:30:08 STEP 1018 H= 254:30 +CPU= 39.151 16:30:48 STEP 1019 H= 254:45 +CPU= 37.329 16:31:37 STEP 1020 H= 255:00 +CPU= 46.137 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001020 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001020 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001020 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.07_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a007 (argv4) batchid: 949 (argv5) wuid: 12166401 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.07 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c7ce91203b1b08443f4e24d55a126930 to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a007_2016092300_15_949_12166401.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a007_2016092300_15_949_12166401.zip ic_ancil_file: ic_ancil_12166401 ifsdata_file: ifsdata_12166401 climate_data_file: clim_data_12166401 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/10/ic_ancil_12166401.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12166401.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_450dacd8598235ff373223ea04e30a3e to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12166401.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12166401.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12166401.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12166401.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12166401.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12166401.xml -- Model is restarting -- Adjusting last_iter, 1021, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12166401.xml last_cpu_time: 36166 upload_file_number: 10 last_iter: 1007 last_upload: 864000 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a007_2016092300_15_949_12166401_0_r583304937 The child process has been launched with process id: 187368 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x30eaa80 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:187368:187368] [20221215:163406:1671082446:0.001] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 16:35:03 STEP 1008 H= 252:00 +CPU= 45.857 16:35:55 STEP 1009 H= 252:15 +CPU= 50.139 16:36:51 STEP 1010 H= 252:30 +CPU= 52.420 16:37:41 STEP 1011 H= 252:45 +CPU= 48.569 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:48:07 STEP 1012 H= 253:00 +CPU= 47.856 16:49:00 STEP 1013 H= 253:15 +CPU= 49.812 16:49:49 STEP 1014 H= 253:30 +CPU= 46.374 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:59:05 STEP 1015 H= 253:45 +CPU= 50.727 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:32 STEP 1016 H= 254:00 +CPU= 47.912 17:13:31 STEP 1017 H= 254:15 +CPU= 49.939 17:14:31 STEP 1018 H= 254:30 +CPU= 51.779 17:15:29 STEP 1019 H= 254:45 +CPU= 50.262 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:20:37 STEP 1020 H= 255:00 +CPU= 58.120 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001020 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001020 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001020 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:27:18 STEP 1021 H= 255:15 +CPU= 42.496 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:36:15 STEP 1022 H= 255:30 +CPU= 46.583 17:36:50 STEP 1023 H= 255:45 +CPU= 30.258 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:49:23 STEP 1024 H= 256:00 +CPU= 31.545 17:50:02 STEP 1025 H= 256:15 +CPU= 32.387 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:56:46 STEP 1026 H= 256:30 +CPU= 25.492 17:57:14 STEP 1027 H= 256:45 +CPU= 26.415 17:57:50 STEP 1028 H= 257:00 +CPU= 34.129 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:00:24 STEP 1029 H= 257:15 +CPU= 30.595 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:02:54 STEP 1030 H= 257:30 +CPU= 26.729 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:06:39 STEP 1031 H= 257:45 +CPU= 27.861 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:20:53 STEP 1032 H= 258:00 +CPU= 33.498 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001032 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001032 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001032 18:21:27 STEP 1033 H= 258:15 +CPU= 30.876 18:22:03 STEP 1034 H= 258:30 +CPU= 32.239 18:22:38 STEP 1035 H= 258:45 +CPU= 31.605 18:23:15 STEP 1036 H= 259:00 +CPU= 33.551 18:23:51 STEP 1037 H= 259:15 +CPU= 31.805 18:24:25 STEP 1038 H= 259:30 +CPU= 30.968 18:25:05 STEP 1039 H= 259:45 +CPU= 35.223 18:25:48 STEP 1040 H= 260:00 +CPU= 38.288 18:26:26 STEP 1041 H= 260:15 +CPU= 34.829 18:27:05 STEP 1042 H= 260:30 +CPU= 36.078 18:27:44 STEP 1043 H= 260:45 +CPU= 35.591 18:28:34 STEP 1044 H= 261:00 +CPU= 44.228 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001044 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001044 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001044 18:29:15 STEP 1045 H= 261:15 +CPU= 37.385 18:29:58 STEP 1046 H= 261:30 +CPU= 38.810 18:30:37 STEP 1047 H= 261:45 +CPU= 34.919 18:31:19 STEP 1048 H= 262:00 +CPU= 38.083 18:31:58 STEP 1049 H= 262:15 +CPU= 35.522 18:32:36 STEP 1050 H= 262:30 +CPU= 35.068 18:33:15 STEP 1051 H= 262:45 +CPU= 35.133 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:36:02 STEP 1052 H= 263:00 +CPU= 36.773 18:36:43 STEP 1053 H= 263:15 +CPU= 36.024 18:37:26 STEP 1054 H= 263:30 +CPU= 38.012 18:38:07 STEP 1055 H= 263:45 +CPU= 36.171 18:39:02 STEP 1056 H= 264:00 +CPU= 50.608 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001056 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001056 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+000960 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+000972 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+000984 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+000996 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001008 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001020 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001032 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001044 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001044 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001044 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a007_2016092300_15_949_12166401_0_r583304937_10.zip Uploading the intermediate file: upload_file_10.zip 18:42:31 STEP 1057 H= 264:15 +CPU= 16.780 18:42:40 STEP 1058 H= 264:30 +CPU= 8.776 18:42:49 STEP 1059 H= 264:45 +CPU= 8.710 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:50:09 STEP 1060 H= 265:00 +CPU= 35.875 18:50:49 STEP 1061 H= 265:15 +CPU= 38.698 18:51:29 STEP 1062 H= 265:30 +CPU= 36.080 18:52:10 STEP 1063 H= 265:45 +CPU= 38.069 18:52:53 STEP 1064 H= 266:00 +CPU= 40.446 18:53:35 STEP 1065 H= 266:15 +CPU= 38.004 18:54:17 STEP 1066 H= 266:30 +CPU= 39.174 18:55:01 STEP 1067 H= 266:45 +CPU= 40.177 18:55:50 STEP 1068 H= 267:00 +CPU= 46.351 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001068 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001068 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001068 18:56:25 STEP 1069 H= 267:15 +CPU= 32.414 18:56:59 STEP 1070 H= 267:30 +CPU= 32.421 18:57:39 STEP 1071 H= 267:45 +CPU= 37.704 18:58:20 STEP 1072 H= 268:00 +CPU= 37.356 18:59:03 STEP 1073 H= 268:15 +CPU= 39.700 18:59:40 STEP 1074 H= 268:30 +CPU= 35.276 19:00:20 STEP 1075 H= 268:45 +CPU= 36.103 19:00:59 STEP 1076 H= 269:00 +CPU= 36.149 19:01:36 STEP 1077 H= 269:15 +CPU= 35.349 19:02:15 STEP 1078 H= 269:30 +CPU= 36.505 19:02:52 STEP 1079 H= 269:45 +CPU= 34.031 19:03:38 STEP 1080 H= 270:00 +CPU= 43.519 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001080 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001080 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001080 19:04:14 STEP 1081 H= 270:15 +CPU= 34.019 19:04:45 STEP 1082 H= 270:30 +CPU= 29.958 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.07_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a007 (argv4) batchid: 949 (argv5) wuid: 12166401 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.07 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c7ce91203b1b08443f4e24d55a126930 to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a007_2016092300_15_949_12166401.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a007_2016092300_15_949_12166401.zip ic_ancil_file: ic_ancil_12166401 ifsdata_file: ifsdata_12166401 climate_data_file: clim_data_12166401 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/10/ic_ancil_12166401.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12166401.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_450dacd8598235ff373223ea04e30a3e to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12166401.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12166401.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12166401.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12166401.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12166401.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12166401.xml -- Model is restarting -- Adjusting last_iter, 1083, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12166401.xml last_cpu_time: 38995 upload_file_number: 11 last_iter: 1055 last_upload: 950400 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a007_2016092300_15_949_12166401_0_r583304937 The child process has been launched with process id: 220590 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x30eaa80 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:191143:1671091903:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:12:24 STEP 1056 H= 264:00 +CPU= 32.675 19:12:56 STEP 1057 H= 264:15 +CPU= 29.654 19:13:30 STEP 1058 H= 264:30 +CPU= 32.387 19:14:06 STEP 1059 H= 264:45 +CPU= 35.349 19:14:34 STEP 1060 H= 265:00 +CPU= 26.600 19:14:58 STEP 1061 H= 265:15 +CPU= 23.316 19:15:23 STEP 1062 H= 265:30 +CPU= 24.958 19:15:48 STEP 1063 H= 265:45 +CPU= 24.590 19:16:12 STEP 1064 H= 266:00 +CPU= 22.950 19:16:34 STEP 1065 H= 266:15 +CPU= 22.184 19:16:57 STEP 1066 H= 266:30 +CPU= 22.181 19:17:20 STEP 1067 H= 266:45 +CPU= 22.346 19:17:50 STEP 1068 H= 267:00 +CPU= 29.541 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001068 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001068 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001068 19:18:15 STEP 1069 H= 267:15 +CPU= 23.414 19:18:39 STEP 1070 H= 267:30 +CPU= 23.123 19:19:03 STEP 1071 H= 267:45 +CPU= 23.446 19:19:24 STEP 1072 H= 268:00 +CPU= 20.657 19:19:47 STEP 1073 H= 268:15 +CPU= 22.014 19:20:09 STEP 1074 H= 268:30 +CPU= 21.956 19:20:35 STEP 1075 H= 268:45 +CPU= 24.366 19:21:00 STEP 1076 H= 269:00 +CPU= 24.433 19:21:26 STEP 1077 H= 269:15 +CPU= 24.984 19:21:49 STEP 1078 H= 269:30 +CPU= 22.396 19:22:12 STEP 1079 H= 269:45 +CPU= 22.379 19:22:42 STEP 1080 H= 270:00 +CPU= 28.804 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001080 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001080 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001080 19:23:02 STEP 1081 H= 270:15 +CPU= 19.666 19:23:25 STEP 1082 H= 270:30 +CPU= 22.004 19:23:48 STEP 1083 H= 270:45 +CPU= 21.983 19:24:15 STEP 1084 H= 271:00 +CPU= 24.958 19:24:37 STEP 1085 H= 271:15 +CPU= 21.843 19:24:57 STEP 1086 H= 271:30 +CPU= 19.703 19:25:19 STEP 1087 H= 271:45 +CPU= 21.541 19:25:40 STEP 1088 H= 272:00 +CPU= 21.443 19:26:06 STEP 1089 H= 272:15 +CPU= 25.122 19:26:26 STEP 1090 H= 272:30 +CPU= 19.854 19:26:47 STEP 1091 H= 272:45 +CPU= 19.880 19:27:13 STEP 1092 H= 273:00 +CPU= 24.900 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001092 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001092 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001092 19:27:34 STEP 1093 H= 273:15 +CPU= 18.983 19:27:53 STEP 1094 H= 273:30 +CPU= 19.209 19:28:13 STEP 1095 H= 273:45 +CPU= 19.237 19:28:33 STEP 1096 H= 274:00 +CPU= 19.780 19:28:55 STEP 1097 H= 274:15 +CPU= 21.036 19:29:16 STEP 1098 H= 274:30 +CPU= 20.788 19:29:42 STEP 1099 H= 274:45 +CPU= 25.386 19:30:16 STEP 1100 H= 275:00 +CPU= 31.613 19:30:46 STEP 1101 H= 275:15 +CPU= 28.602 19:31:16 STEP 1102 H= 275:30 +CPU= 28.281 19:31:40 STEP 1103 H= 275:45 +CPU= 23.114 19:32:10 STEP 1104 H= 276:00 +CPU= 29.262 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001104 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001104 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001104 19:32:34 STEP 1105 H= 276:15 +CPU= 23.840 19:33:00 STEP 1106 H= 276:30 +CPU= 25.393 19:33:28 STEP 1107 H= 276:45 +CPU= 27.637 19:33:53 STEP 1108 H= 277:00 +CPU= 24.217 19:34:17 STEP 1109 H= 277:15 +CPU= 23.846 19:34:45 STEP 1110 H= 277:30 +CPU= 27.190 19:35:11 STEP 1111 H= 277:45 +CPU= 25.504 19:35:37 STEP 1112 H= 278:00 +CPU= 25.755 19:36:02 STEP 1113 H= 278:15 +CPU= 23.574 19:36:33 STEP 1114 H= 278:30 +CPU= 29.882 19:37:04 STEP 1115 H= 278:45 +CPU= 28.811 19:37:33 STEP 1116 H= 279:00 +CPU= 28.264 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001116 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001116 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001116 19:37:57 STEP 1117 H= 279:15 +CPU= 23.526 19:38:24 STEP 1118 H= 279:30 +CPU= 26.042 19:38:53 STEP 1119 H= 279:45 +CPU= 26.609 19:39:24 STEP 1120 H= 280:00 +CPU= 29.875 19:39:50 STEP 1121 H= 280:15 +CPU= 24.400 19:40:16 STEP 1122 H= 280:30 +CPU= 24.932 19:40:45 STEP 1123 H= 280:45 +CPU= 27.061 19:41:15 STEP 1124 H= 281:00 +CPU= 29.242 19:41:42 STEP 1125 H= 281:15 +CPU= 26.417 19:42:12 STEP 1126 H= 281:30 +CPU= 29.317 19:42:40 STEP 1127 H= 281:45 +CPU= 27.608 19:43:12 STEP 1128 H= 282:00 +CPU= 29.743 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001128 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001128 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001128 19:43:35 STEP 1129 H= 282:15 +CPU= 21.385 19:44:05 STEP 1130 H= 282:30 +CPU= 29.309 19:44:32 STEP 1131 H= 282:45 +CPU= 25.861 19:45:01 STEP 1132 H= 283:00 +CPU= 28.457 19:45:29 STEP 1133 H= 283:15 +CPU= 26.347 19:45:55 STEP 1134 H= 283:30 +CPU= 24.835 19:46:23 STEP 1135 H= 283:45 +CPU= 27.252 19:46:54 STEP 1136 H= 284:00 +CPU= 28.813 19:47:23 STEP 1137 H= 284:15 +CPU= 28.234 19:47:50 STEP 1138 H= 284:30 +CPU= 26.193 19:48:14 STEP 1139 H= 284:45 +CPU= 22.746 19:48:51 STEP 1140 H= 285:00 +CPU= 35.119 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001140 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001140 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001140 19:49:19 STEP 1141 H= 285:15 +CPU= 26.184 19:49:47 STEP 1142 H= 285:30 +CPU= 27.501 19:50:14 STEP 1143 H= 285:45 +CPU= 24.826 19:50:40 STEP 1144 H= 286:00 +CPU= 24.364 19:51:03 STEP 1145 H= 286:15 +CPU= 22.375 19:51:29 STEP 1146 H= 286:30 +CPU= 24.102 19:51:58 STEP 1147 H= 286:45 +CPU= 27.485 19:52:27 STEP 1148 H= 287:00 +CPU= 26.824 19:52:53 STEP 1149 H= 287:15 +CPU= 24.775 19:53:20 STEP 1150 H= 287:30 +CPU= 25.416 19:53:47 STEP 1151 H= 287:45 +CPU= 26.398 19:54:27 STEP 1152 H= 288:00 +CPU= 37.897 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001152 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001152 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMGGh7zg+001140 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMSHh7zg+001140 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401/ICMUAh7zg+001140 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a007_2016092300_15_949_12166401_0_r583304937_11.zip Uploading the intermediate file: upload_file_11.zip 19:54:51 STEP 1153 H= 288:15 +CPU= 22.968 19:55:17 STEP 1154 H= 288:30 +CPU= 25.238 19:55:43 STEP 1155 H= 288:45 +CPU= 25.602 19:56:11 STEP 1156 H= 289:00 +CPU= 27.132 19:56:38 STEP 1157 H= 289:15 +CPU= 26.241 19:57:05 STEP 1158 H= 289:30 +CPU= 25.368 19:57:35 STEP 1159 H= 289:45 +CPU= 27.985 19:58:03 STEP 1160 H= 290:00 +CPU= 25.908 19:58:29 STEP 1161 H= 290:15 +CPU= 25.866 19:59:01 STEP 1162 H= 290:30 +CPU= 31.067 19:59:29 STEP 1163 H= 290:45 +CPU= 27.254 20:00:03 STEP 1164 H= 291:00 +CPU= 32.481 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001164 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001164 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001164 20:00:33 STEP 1165 H= 291:15 +CPU= 28.452 20:01:00 STEP 1166 H= 291:30 +CPU= 26.311 20:01:26 STEP 1167 H= 291:45 +CPU= 24.564 20:01:50 STEP 1168 H= 292:00 +CPU= 23.678 20:02:15 STEP 1169 H= 292:15 +CPU= 24.181 20:02:39 STEP 1170 H= 292:30 +CPU= 23.382 20:03:04 STEP 1171 H= 292:45 +CPU= 23.990 20:03:30 STEP 1172 H= 293:00 +CPU= 25.081 20:03:53 STEP 1173 H= 293:15 +CPU= 21.569 20:04:14 STEP 1174 H= 293:30 +CPU= 20.041 20:04:39 STEP 1175 H= 293:45 +CPU= 24.841 20:05:10 STEP 1176 H= 294:00 +CPU= 29.591 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001176 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001176 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001176 20:05:36 STEP 1177 H= 294:15 +CPU= 25.196 20:06:01 STEP 1178 H= 294:30 +CPU= 25.184 20:06:24 STEP 1179 H= 294:45 +CPU= 22.476 20:06:50 STEP 1180 H= 295:00 +CPU= 24.216 20:07:17 STEP 1181 H= 295:15 +CPU= 25.364 20:07:41 STEP 1182 H= 295:30 +CPU= 23.432 20:08:07 STEP 1183 H= 295:45 +CPU= 24.824 20:08:33 STEP 1184 H= 296:00 +CPU= 24.732 20:08:57 STEP 1185 H= 296:15 +CPU= 22.794 20:09:22 STEP 1186 H= 296:30 +CPU= 23.701 20:09:45 STEP 1187 H= 296:45 +CPU= 21.143 20:10:14 STEP 1188 H= 297:00 +CPU= 28.410 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001188 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001188 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001188 20:10:37 STEP 1189 H= 297:15 +CPU= 21.305 20:11:00 STEP 1190 H= 297:30 +CPU= 21.982 20:11:24 STEP 1191 H= 297:45 +CPU= 22.258 20:11:49 STEP 1192 H= 298:00 +CPU= 23.828 20:12:13 STEP 1193 H= 298:15 +CPU= 22.574 20:12:43 STEP 1194 H= 298:30 +CPU= 28.348 Suspend request received from the BOINC client, suspending the child process [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5397MB (heap), 4897MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x80f380 (old at (nil)) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3843] 5397 MB (maxheap), 4897 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : CALL_SL [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004) [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : LAPINEA [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201453:1671095693:3789.961] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:3897] : LARMES [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201454:1671095694:3790.962] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:roo:1:1:220590:220590] [20221215:201454:1671095694:3790.962] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 000000000080FFB1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8CDC0 Unknown Unknown Unknown oifs_43r3_model.e 0000000001D5CDD8 Unknown Unknown Unknown oifs_43r3_model.e 00000000011C7B12 larmes_ 543 larmes.F90 oifs_43r3_model.e 00000000011C319D lapinea_ 270 lapinea.F90 oifs_43r3_model.e 00000000011BB1CC call_sl_ 300 call_sl.F90 oifs_43r3_model.e 000000000139E65D call_sl_heap_ 94 call_sl_heap.F90 oifs_43r3_model.e 0000000000D53278 gp_model_ 533 gp_model.F90 oifs_43r3_model.e 0000000000D3FB32 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 00000000014A1C3E scan2m_ 535 scan2m.F90 oifs_43r3_model.e 00000000006AD494 stepo_ 327 stepo.F90 oifs_43r3_model.e 00000000005750DC cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 000000000055E289 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 000000000055D452 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000546AC8 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040276F MAIN__ 96 master.F90 oifs_43r3_model.e 000000000040270D Unknown Unknown Unknown oifs_43r3_model.e 0000000001D83B1A Unknown Unknown Unknown oifs_43r3_model.e 0000000001D85377 Unknown Unknown Unknown oifs_43r3_model.e 00000000004025E5 Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.07_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a007 (argv4) batchid: 949 (argv5) wuid: 12166401 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/10 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.07 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12166401 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_app_1.07_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_c7ce91203b1b08443f4e24d55a126930 to: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a007_2016092300_15_949_12166401.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/10/oifs_43r3_bl_a007_2016092300_15_949_12166401.zip ic_ancil_file: ic_ancil_12166401 ifsdata_file: ifsdata_12166401 climate_data_file: clim_data_12166401 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/10/ic_ancil_12166401.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/10/ic_ancil_12166401.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_450dacd8598235ff373223ea04e30a3e to: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12166401.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/10/ifsdata/ifsdata_12166401.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/10/159l_2/clim_data_12166401.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/10/159l_2/clim_data_12166401.zip Checking for progress XML file: /var/lib/boinc-client/slots/10/progress_file_12166401.xml Opened progress file ok : /var/lib/boinc-client/slots/10/progress_file_12166401.xml -- Model is restarting -- Adjusting last_iter, 1195, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/10/progress_file_12166401.xml last_cpu_time: 42515 upload_file_number: 12 last_iter: 1151 last_upload: 1036800 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a007_2016092300_15_949_12166401_0_r583304937 The child process has been launched with process id: 6290 Executing the command: /var/lib/boinc-client/slots/10/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x30eaa80 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x80f7e0 (old at 0x401140) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x80f7e0 (old at (nil)) [EC_DRHOOK:roo:1:1:6290:6290] [20221216:112012:1671150012:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 11:20:39 STEP 1152 H= 288:00 +CPU= 23.043 11:21:17 STEP 1153 H= 288:15 +CPU= 36.690 11:21:51 STEP 1154 H= 288:30 +CPU= 32.667 11:22:28 STEP 1155 H= 288:45 +CPU= 34.972 11:23:00 STEP 1156 H= 289:00 +CPU= 30.823 11:23:38 STEP 1157 H= 289:15 +CPU= 36.842 11:24:09 STEP 1158 H= 289:30 +CPU= 30.843 11:24:39 STEP 1159 H= 289:45 +CPU= 28.958 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:27:14 STEP 1160 H= 290:00 +CPU= 27.405 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:32:21 STEP 1161 H= 290:15 +CPU= 33.853 11:32:55 STEP 1162 H= 290:30 +CPU= 31.514 11:33:25 STEP 1163 H= 290:45 +CPU= 28.651 11:34:07 STEP 1164 H= 291:00 +CPU= 38.861 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001164 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001164 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001164 11:34:43 STEP 1165 H= 291:15 +CPU= 33.253 11:35:19 STEP 1166 H= 291:30 +CPU= 33.408 11:35:55 STEP 1167 H= 291:45 +CPU= 33.381 11:36:25 STEP 1168 H= 292:00 +CPU= 27.862 11:36:59 STEP 1169 H= 292:15 +CPU= 31.183 11:37:37 STEP 1170 H= 292:30 +CPU= 34.725 11:38:15 STEP 1171 H= 292:45 +CPU= 36.045 11:38:52 STEP 1172 H= 293:00 +CPU= 34.167 11:39:27 STEP 1173 H= 293:15 +CPU= 32.733 11:40:05 STEP 1174 H= 293:30 +CPU= 35.284 11:40:39 STEP 1175 H= 293:45 +CPU= 32.162 11:41:22 STEP 1176 H= 294:00 +CPU= 40.509 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001176 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001176 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001176 11:41:58 STEP 1177 H= 294:15 +CPU= 33.632 11:42:34 STEP 1178 H= 294:30 +CPU= 34.605 11:43:09 STEP 1179 H= 294:45 +CPU= 34.381 11:43:44 STEP 1180 H= 295:00 +CPU= 34.075 11:44:20 STEP 1181 H= 295:15 +CPU= 34.439 11:44:53 STEP 1182 H= 295:30 +CPU= 31.441 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:47:43 STEP 1183 H= 295:45 +CPU= 31.381 11:48:19 STEP 1184 H= 296:00 +CPU= 32.991 11:48:53 STEP 1185 H= 296:15 +CPU= 31.913 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:51:29 STEP 1186 H= 296:30 +CPU= 32.637 11:52:05 STEP 1187 H= 296:45 +CPU= 32.959 11:52:50 STEP 1188 H= 297:00 +CPU= 41.488 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001188 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001188 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001188 11:53:29 STEP 1189 H= 297:15 +CPU= 36.267 11:54:03 STEP 1190 H= 297:30 +CPU= 30.675 11:54:41 STEP 1191 H= 297:45 +CPU= 35.377 11:55:20 STEP 1192 H= 298:00 +CPU= 36.282 11:55:54 STEP 1193 H= 298:15 +CPU= 32.194 11:56:29 STEP 1194 H= 298:30 +CPU= 32.076 11:57:06 STEP 1195 H= 298:45 +CPU= 34.159 11:57:42 STEP 1196 H= 299:00 +CPU= 34.745 11:58:18 STEP 1197 H= 299:15 +CPU= 33.926 11:58:58 STEP 1198 H= 299:30 +CPU= 36.188 11:59:34 STEP 1199 H= 299:45 +CPU= 34.071 12:00:22 STEP 1200 H= 300:00 +CPU= 45.344 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMGGh7zg+001200 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMSHh7zg+001200 Moving to projects directory: /var/lib/boinc-client/slots/10/ICMUAh7zg+001200 12:01:02 STEP 1201 H= 300:15 +CPU= 37.206 12:01:38 STEP 1202 H= 300:30 +CPU= 33.011 Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org