Name | oifs_43r3_ps_0321_2014050100_123_983_12199965_0 |
Workunit | 12199965 |
Created | 24 Dec 2022, 9:52:34 UTC |
Sent | 29 Dec 2022, 18:36:34 UTC |
Report deadline | 28 Jan 2023, 18:36:34 UTC |
Received | 1 Jan 2023, 8:34:33 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1511650 |
Run time | 1 days 13 hours 59 min 27 sec |
CPU time | 1 days 8 hours 15 min 30 sec |
Validate state | Invalid |
Credit | 4,838.40 |
Device peak FLOPS | 2.42 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,419.01 MB |
Peak swap size | 4,978.13 MB |
Peak disk usage | 7,590.94 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> 04:20 STEP 1073 H=1073:00 +CPU= 68.775 20:06:32 STEP 1074 H=1074:00 +CPU=114.572 20:07:51 STEP 1075 H=1075:00 +CPU= 68.530 20:09:11 STEP 1076 H=1076:00 +CPU= 71.057 20:11:19 STEP 1077 H=1077:00 +CPU=111.434 20:12:44 STEP 1078 H=1078:00 +CPU= 71.472 20:14:06 STEP 1079 H=1079:00 +CPU= 71.618 20:16:39 STEP 1080 H=1080:00 +CPU=130.738 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001080 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001080 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001056 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001068 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001068 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_44.zip Uploading the intermediate file: upload_file_44.zip 20:18:01 STEP 1081 H=1081:00 +CPU= 69.096 20:19:24 STEP 1082 H=1082:00 +CPU= 71.795 20:21:35 STEP 1083 H=1083:00 +CPU=112.016 20:22:59 STEP 1084 H=1084:00 +CPU= 71.795 20:24:17 STEP 1085 H=1085:00 +CPU= 68.933 20:26:30 STEP 1086 H=1086:00 +CPU=115.148 20:27:48 STEP 1087 H=1087:00 +CPU= 69.152 20:29:09 STEP 1088 H=1088:00 +CPU= 72.081 20:31:20 STEP 1089 H=1089:00 +CPU=112.224 20:32:42 STEP 1090 H=1090:00 +CPU= 71.753 20:34:04 STEP 1091 H=1091:00 +CPU= 69.045 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2014050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0321 (argv4) batchid: 983 (argv5) wuid: 12199965 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip ic_ancil_file: ic_ancil_12199965 ifsdata_file: ifsdata_12199965 climate_data_file: clim_data_12199965 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml -- Model is restarting -- Adjusting last_iter, 1092, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml last_cpu_time: 101990 upload_file_number: 45 last_iter: 1079 last_upload: 3888000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732 The child process has been launched with process id: 1277610 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1277610:1277610] [20221231:214145:1672522905:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 21:44:14 STEP 1080 H=1080:00 +CPU=126.610 21:45:34 STEP 1081 H=1081:00 +CPU= 68.614 21:46:56 STEP 1082 H=1082:00 +CPU= 71.349 21:48:59 STEP 1083 H=1083:00 +CPU=110.733 21:50:22 STEP 1084 H=1084:00 +CPU= 71.495 21:51:40 STEP 1085 H=1085:00 +CPU= 68.509 21:53:49 STEP 1086 H=1086:00 +CPU=114.404 21:55:10 STEP 1087 H=1087:00 +CPU= 69.124 21:56:32 STEP 1088 H=1088:00 +CPU= 71.774 21:58:45 STEP 1089 H=1089:00 +CPU=112.297 22:00:11 STEP 1090 H=1090:00 +CPU= 72.019 22:01:34 STEP 1091 H=1091:00 +CPU= 69.361 22:04:01 STEP 1092 H=1092:00 +CPU=125.346 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001092 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001092 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001092 22:05:22 STEP 1093 H=1093:00 +CPU= 69.186 22:06:42 STEP 1094 H=1094:00 +CPU= 71.741 22:08:51 STEP 1095 H=1095:00 +CPU=112.172 22:10:14 STEP 1096 H=1096:00 +CPU= 71.934 22:11:33 STEP 1097 H=1097:00 +CPU= 69.016 22:13:45 STEP 1098 H=1098:00 +CPU=114.569 22:15:04 STEP 1099 H=1099:00 +CPU= 68.785 Suspend request received from the BOINC client, suspending the child process Resuming the child process 22:16:40 STEP 1100 H=1100:00 +CPU= 71.593 22:18:48 STEP 1101 H=1101:00 +CPU=111.326 22:20:08 STEP 1102 H=1102:00 +CPU= 71.447 22:21:29 STEP 1103 H=1103:00 +CPU= 71.146 22:23:57 STEP 1104 H=1104:00 +CPU=129.718 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001104 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001104 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001080 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001092 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001092 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_45.zip Uploading the intermediate file: upload_file_45.zip 22:25:17 STEP 1105 H=1105:00 +CPU= 68.479 22:26:40 STEP 1106 H=1106:00 +CPU= 71.438 22:28:49 STEP 1107 H=1107:00 +CPU=111.583 22:30:12 STEP 1108 H=1108:00 +CPU= 71.387 22:31:35 STEP 1109 H=1109:00 +CPU= 69.092 22:33:51 STEP 1110 H=1110:00 +CPU=114.920 22:35:10 STEP 1111 H=1111:00 +CPU= 68.852 22:36:35 STEP 1112 H=1112:00 +CPU= 71.677 22:38:43 STEP 1113 H=1113:00 +CPU=111.919 22:40:06 STEP 1114 H=1114:00 +CPU= 72.022 22:41:27 STEP 1115 H=1115:00 +CPU= 69.181 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2014050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0321 (argv4) batchid: 983 (argv5) wuid: 12199965 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip ic_ancil_file: ic_ancil_12199965 ifsdata_file: ifsdata_12199965 climate_data_file: clim_data_12199965 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml -- Model is restarting -- Adjusting last_iter, 1116, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml last_cpu_time: 105091 upload_file_number: 46 last_iter: 1103 last_upload: 3974400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732 The child process has been launched with process id: 1282947 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1282947:1282947] [20221231:233148:1672529508:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 23:34:20 STEP 1104 H=1104:00 +CPU=127.004 23:35:39 STEP 1105 H=1105:00 +CPU= 68.733 23:37:01 STEP 1106 H=1106:00 +CPU= 71.374 23:39:12 STEP 1107 H=1107:00 +CPU=111.489 23:40:35 STEP 1108 H=1108:00 +CPU= 71.736 23:41:56 STEP 1109 H=1109:00 +CPU= 68.959 23:44:10 STEP 1110 H=1110:00 +CPU=114.942 23:45:32 STEP 1111 H=1111:00 +CPU= 69.162 23:46:55 STEP 1112 H=1112:00 +CPU= 71.795 23:49:08 STEP 1113 H=1113:00 +CPU=112.110 23:50:31 STEP 1114 H=1114:00 +CPU= 71.938 23:51:50 STEP 1115 H=1115:00 +CPU= 68.948 23:54:10 STEP 1116 H=1116:00 +CPU=125.098 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001116 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001116 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001116 23:55:32 STEP 1117 H=1117:00 +CPU= 68.934 23:56:55 STEP 1118 H=1118:00 +CPU= 71.430 23:59:03 STEP 1119 H=1119:00 +CPU=111.490 Suspend request received from the BOINC client, suspending the child process Resuming the child process 00:01:30 STEP 1120 H=1120:00 +CPU= 71.545 00:02:48 STEP 1121 H=1121:00 +CPU= 68.449 Suspend request received from the BOINC client, suspending the child process Resuming the child process 00:07:04 STEP 1122 H=1122:00 +CPU=114.705 00:08:29 STEP 1123 H=1123:00 +CPU= 68.973 Suspend request received from the BOINC client, suspending the child process Resuming the child process 00:10:55 STEP 1124 H=1124:00 +CPU= 71.602 00:13:05 STEP 1125 H=1125:00 +CPU=111.795 Suspend request received from the BOINC client, suspending the child process Resuming the child process 00:15:40 STEP 1126 H=1126:00 +CPU= 72.016 00:17:05 STEP 1127 H=1127:00 +CPU= 71.438 00:19:44 STEP 1128 H=1128:00 +CPU=130.999 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001128 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001128 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001104 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001116 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001116 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_46.zip Uploading the intermediate file: upload_file_46.zip 00:21:06 STEP 1129 H=1129:00 +CPU= 68.934 00:22:29 STEP 1130 H=1130:00 +CPU= 71.604 Suspend request received from the BOINC client, suspending the child process Resuming the child process 00:26:41 STEP 1131 H=1131:00 +CPU=112.189 00:28:03 STEP 1132 H=1132:00 +CPU= 71.528 00:29:22 STEP 1133 H=1133:00 +CPU= 68.799 00:31:32 STEP 1134 H=1134:00 +CPU=114.664 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2014050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0321 (argv4) batchid: 983 (argv5) wuid: 12199965 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip ic_ancil_file: ic_ancil_12199965 ifsdata_file: ifsdata_12199965 climate_data_file: clim_data_12199965 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml -- Model is restarting -- Adjusting last_iter, 1135, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml last_cpu_time: 107824 upload_file_number: 47 last_iter: 1127 last_upload: 4060800 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732 The child process has been launched with process id: 1289273 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1289273:1289273] [20230101:013320:1672536800:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 01:35:55 STEP 1128 H=1128:00 +CPU=128.528 01:37:15 STEP 1129 H=1129:00 +CPU= 69.386 01:38:38 STEP 1130 H=1130:00 +CPU= 72.139 01:40:52 STEP 1131 H=1131:00 +CPU=112.774 01:42:16 STEP 1132 H=1132:00 +CPU= 72.069 01:43:38 STEP 1133 H=1133:00 +CPU= 69.339 01:45:54 STEP 1134 H=1134:00 +CPU=116.032 01:47:15 STEP 1135 H=1135:00 +CPU= 69.879 01:48:41 STEP 1136 H=1136:00 +CPU= 72.683 01:50:46 STEP 1137 H=1137:00 +CPU=112.591 01:52:11 STEP 1138 H=1138:00 +CPU= 72.677 01:53:32 STEP 1139 H=1139:00 +CPU= 69.986 01:56:00 STEP 1140 H=1140:00 +CPU=126.149 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001140 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001140 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001140 01:57:24 STEP 1141 H=1141:00 +CPU= 69.990 01:58:50 STEP 1142 H=1142:00 +CPU= 72.742 02:01:04 STEP 1143 H=1143:00 +CPU=113.497 02:02:28 STEP 1144 H=1144:00 +CPU= 72.642 02:03:47 STEP 1145 H=1145:00 +CPU= 69.701 02:06:02 STEP 1146 H=1146:00 +CPU=116.521 02:07:22 STEP 1147 H=1147:00 +CPU= 69.715 02:08:47 STEP 1148 H=1148:00 +CPU= 72.547 02:10:58 STEP 1149 H=1149:00 +CPU=112.880 02:12:22 STEP 1150 H=1150:00 +CPU= 72.668 02:13:46 STEP 1151 H=1151:00 +CPU= 72.434 02:16:17 STEP 1152 H=1152:00 +CPU=132.394 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001152 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001152 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001128 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001140 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001140 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001140 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_47.zip Uploading the intermediate file: upload_file_47.zip 02:17:41 STEP 1153 H=1153:00 +CPU= 70.119 02:19:03 STEP 1154 H=1154:00 +CPU= 72.736 02:21:12 STEP 1155 H=1155:00 +CPU=113.221 02:22:36 STEP 1156 H=1156:00 +CPU= 72.686 02:23:52 STEP 1157 H=1157:00 +CPU= 69.699 02:26:06 STEP 1158 H=1158:00 +CPU=116.132 02:27:28 STEP 1159 H=1159:00 +CPU= 70.041 02:28:52 STEP 1160 H=1160:00 +CPU= 72.865 02:31:05 STEP 1161 H=1161:00 +CPU=113.531 02:32:32 STEP 1162 H=1162:00 +CPU= 72.911 02:33:51 STEP 1163 H=1163:00 +CPU= 70.279 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2014050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0321 (argv4) batchid: 983 (argv5) wuid: 12199965 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip ic_ancil_file: ic_ancil_12199965 ifsdata_file: ifsdata_12199965 climate_data_file: clim_data_12199965 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml -- Model is restarting -- Adjusting last_iter, 1164, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml last_cpu_time: 110965 upload_file_number: 48 last_iter: 1151 last_upload: 4147200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732 The child process has been launched with process id: 1295296 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1295296:1295296] [20230101:033710:1672544230:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 03:39:45 STEP 1152 H=1152:00 +CPU=128.512 03:41:07 STEP 1153 H=1153:00 +CPU= 69.646 03:42:31 STEP 1154 H=1154:00 +CPU= 72.417 03:44:44 STEP 1155 H=1155:00 +CPU=113.041 03:46:11 STEP 1156 H=1156:00 +CPU= 72.578 03:47:33 STEP 1157 H=1157:00 +CPU= 69.595 03:49:48 STEP 1158 H=1158:00 +CPU=116.160 03:51:08 STEP 1159 H=1159:00 +CPU= 69.575 03:52:31 STEP 1160 H=1160:00 +CPU= 72.422 03:54:42 STEP 1161 H=1161:00 +CPU=112.878 03:56:08 STEP 1162 H=1162:00 +CPU= 72.129 03:57:30 STEP 1163 H=1163:00 +CPU= 69.430 03:59:51 STEP 1164 H=1164:00 +CPU=125.303 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001164 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001164 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001164 Suspend request received from the BOINC client, suspending the child process Resuming the child process 04:01:24 STEP 1165 H=1165:00 +CPU= 69.702 04:02:49 STEP 1166 H=1166:00 +CPU= 72.557 04:05:02 STEP 1167 H=1167:00 +CPU=112.958 04:06:23 STEP 1168 H=1168:00 +CPU= 72.186 04:07:44 STEP 1169 H=1169:00 +CPU= 69.322 04:09:59 STEP 1170 H=1170:00 +CPU=115.940 04:11:22 STEP 1171 H=1171:00 +CPU= 69.534 04:12:46 STEP 1172 H=1172:00 +CPU= 72.334 04:14:58 STEP 1173 H=1173:00 +CPU=112.600 04:16:24 STEP 1174 H=1174:00 +CPU= 72.335 04:17:50 STEP 1175 H=1175:00 +CPU= 72.342 04:20:21 STEP 1176 H=1176:00 +CPU=131.611 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001176 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001176 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001176 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001152 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMGGhq0f+001164 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMSHhq0f+001164 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965/ICMUAhq0f+001164 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732_48.zip Uploading the intermediate file: upload_file_48.zip 04:21:43 STEP 1177 H=1177:00 +CPU= 69.447 04:23:06 STEP 1178 H=1178:00 +CPU= 72.523 04:25:15 STEP 1179 H=1179:00 +CPU=112.703 04:26:39 STEP 1180 H=1180:00 +CPU= 72.197 04:28:01 STEP 1181 H=1181:00 +CPU= 69.579 04:30:14 STEP 1182 H=1182:00 +CPU=115.585 04:31:35 STEP 1183 H=1183:00 +CPU= 69.297 04:33:00 STEP 1184 H=1184:00 +CPU= 72.177 Suspend request received from the BOINC client, suspending the child process Resuming the child process 04:35:44 STEP 1185 H=1185:00 +CPU=112.101 04:37:13 STEP 1186 H=1186:00 +CPU= 72.748 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2014050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0321 (argv4) batchid: 983 (argv5) wuid: 12199965 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199965 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3d089491fd72531f90fe19f50bd9a3ba to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0321_2014050100_123_983_12199965.zip ic_ancil_file: ic_ancil_12199965 ifsdata_file: ifsdata_12199965 climate_data_file: clim_data_12199965 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_e40cc7015937ba69e8d183d75f61ef90 to: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12199965.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e4329264a92645d668fc829904292540 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12199965.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12199965.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12199965.xml -- Model is restarting -- Adjusting last_iter, 1187, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12199965.xml last_cpu_time: 114034 upload_file_number: 49 last_iter: 1175 last_upload: 4233600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0321_2014050100_123_983_12199965_0_r1568412732 The child process has been launched with process id: 1306053 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:uranus:1:1:1306053:1306053] [20230101:065254:1672555974:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 06:56:34 STEP 1176 H=1176:00 +CPU=126.717 06:57:49 STEP 1177 H=1177:00 +CPU= 68.580 06:59:11 STEP 1178 H=1178:00 +CPU= 71.400 07:01:21 STEP 1179 H=1179:00 +CPU=111.671 07:02:46 STEP 1180 H=1180:00 +CPU= 71.550 07:04:03 STEP 1181 H=1181:00 +CPU= 68.887 07:06:15 STEP 1182 H=1182:00 +CPU=114.518 07:07:35 STEP 1183 H=1183:00 +CPU= 68.778 07:08:58 STEP 1184 H=1184:00 +CPU= 71.686 07:11:07 STEP 1185 H=1185:00 +CPU=111.922 07:12:30 STEP 1186 H=1186:00 +CPU= 71.882 07:13:53 STEP 1187 H=1187:00 +CPU= 68.913 07:16:19 STEP 1188 H=1188:00 +CPU=124.903 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+001188 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+001188 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+001188 07:17:38 STEP 1189 H=1189:00 +CPU= 69.038 07:19:00 STEP 1190 H=1190:00 +CPU= 71.778 07:21:07 STEP 1191 H=1191:00 +CPU=111.629 07:22:27 STEP 1192 H=1192:00 +CPU= 71.586 07:23:47 STEP 1193 H=1193:00 +CPU= 68.798 07:25:56 STEP 1194 H=1194:00 +CPU=114.075 07:27:15 STEP 1195 H=1195:00 +CPU= 68.547 07:28:36 STEP 1196 H=1196:00 +CPU= 71.327 07:30:44 STEP 1197 H=1197:00 +CPU=111.396 07:32:03 STEP 1198 H=1198:00 +CPU= 71.367 07:33:26 STEP 1199 H=1199:00 +CPU= 71.275 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
09 May 2023 16:03:25 | 1511650 | 22286174 | oifs_43r3_ps_0321_2014050100_123_983_12199965_0 | 3,456,000 | 87,109 | 0.0252 |
09 May 2023 16:03:06 | 1511650 | 22286174 | oifs_43r3_ps_0321_2014050100_123_983_12199965_0 | 2,592,000 | 65,151 | 0.0251 |
09 May 2023 16:02:42 | 1511650 | 22286174 | oifs_43r3_ps_0321_2014050100_123_983_12199965_0 | 1,728,000 | 43,162 | 0.0250 |
09 May 2023 16:02:19 | 1511650 | 22286174 | oifs_43r3_ps_0321_2014050100_123_983_12199965_0 | 864,000 | 21,341 | 0.0247 |
©2024 cpdn.org