Name | oifs_43r3_ps_0225_2006050100_123_975_12191869_0 |
Workunit | 12191869 |
Created | 23 Dec 2022, 12:38:03 UTC |
Sent | 27 Dec 2022, 22:19:58 UTC |
Report deadline | 26 Jan 2023, 22:19:58 UTC |
Received | 2 Jan 2023, 23:07:40 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1532160 |
Run time | 2 hours 32 min 35 sec |
CPU time | 1 hours 27 min 10 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 8.21 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,408.30 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 3,578.94 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> HOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72849:72849] [20230102:143751:1672695471:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 14:38:41 STEP 144 H= 144:00 +CPU= 20.564 14:38:54 STEP 145 H= 145:00 +CPU= 11.546 14:39:06 STEP 146 H= 146:00 +CPU= 11.940 14:39:24 STEP 147 H= 147:00 +CPU= 17.731 14:39:36 STEP 148 H= 148:00 +CPU= 11.383 14:39:47 STEP 149 H= 149:00 +CPU= 11.130 14:40:05 STEP 150 H= 150:00 +CPU= 18.335 14:40:16 STEP 151 H= 151:00 +CPU= 11.232 14:40:28 STEP 152 H= 152:00 +CPU= 11.705 14:40:47 STEP 153 H= 153:00 +CPU= 18.321 14:40:58 STEP 154 H= 154:00 +CPU= 11.520 14:41:10 STEP 155 H= 155:00 +CPU= 11.616 14:41:34 STEP 156 H= 156:00 +CPU= 20.889 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000156 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000156 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000156 14:41:45 STEP 157 H= 157:00 +CPU= 11.220 14:41:57 STEP 158 H= 158:00 +CPU= 11.451 14:42:15 STEP 159 H= 159:00 +CPU= 17.845 14:42:27 STEP 160 H= 160:00 +CPU= 11.631 14:42:38 STEP 161 H= 161:00 +CPU= 11.194 14:42:56 STEP 162 H= 162:00 +CPU= 18.129 14:43:07 STEP 163 H= 163:00 +CPU= 11.373 14:43:19 STEP 164 H= 164:00 +CPU= 11.623 14:43:37 STEP 165 H= 165:00 +CPU= 17.978 14:43:49 STEP 166 H= 166:00 +CPU= 12.005 14:44:02 STEP 167 H= 167:00 +CPU= 11.919 14:44:31 STEP 168 H= 168:00 +CPU= 21.980 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000168 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000168 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000144 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000144 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000144 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000156 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000156 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000156 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096_5.zip 14:44:43 STEP 169 H= 169:00 +CPU= 11.528 Uploading the intermediate file: upload_file_5.zip 14:44:55 STEP 170 H= 170:00 +CPU= 12.084 14:45:13 STEP 171 H= 171:00 +CPU= 18.386 14:45:25 STEP 172 H= 172:00 +CPU= 11.898 14:45:37 STEP 173 H= 173:00 +CPU= 11.717 14:45:55 STEP 174 H= 174:00 +CPU= 18.716 14:46:08 STEP 175 H= 175:00 +CPU= 11.491 14:46:20 STEP 176 H= 176:00 +CPU= 11.818 14:46:38 STEP 177 H= 177:00 +CPU= 18.329 14:46:50 STEP 178 H= 178:00 +CPU= 11.673 14:47:02 STEP 179 H= 179:00 +CPU= 11.678 14:47:22 STEP 180 H= 180:00 +CPU= 20.396 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000180 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000180 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000180 14:47:34 STEP 181 H= 181:00 +CPU= 11.568 14:47:46 STEP 182 H= 182:00 +CPU= 11.925 14:48:04 STEP 183 H= 183:00 +CPU= 18.364 14:48:16 STEP 184 H= 184:00 +CPU= 11.907 14:48:28 STEP 185 H= 185:00 +CPU= 11.610 14:48:46 STEP 186 H= 186:00 +CPU= 18.805 14:48:58 STEP 187 H= 187:00 +CPU= 11.286 14:49:13 STEP 188 H= 188:00 +CPU= 11.760 14:49:30 STEP 189 H= 189:00 +CPU= 17.746 14:49:42 STEP 190 H= 190:00 +CPU= 11.276 14:49:53 STEP 191 H= 191:00 +CPU= 11.531 14:50:18 STEP 192 H= 192:00 +CPU= 21.275 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000192 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000192 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000180 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000180 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000180 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096_6.zip Uploading the intermediate file: upload_file_6.zip 14:50:29 STEP 193 H= 193:00 +CPU= 11.372 14:50:41 STEP 194 H= 194:00 +CPU= 11.767 14:50:59 STEP 195 H= 195:00 +CPU= 17.597 14:51:10 STEP 196 H= 196:00 +CPU= 11.526 14:51:26 STEP 197 H= 197:00 +CPU= 11.544 14:51:45 STEP 198 H= 198:00 +CPU= 18.702 14:51:56 STEP 199 H= 199:00 +CPU= 11.026 14:52:08 STEP 200 H= 200:00 +CPU= 11.542 14:52:26 STEP 201 H= 201:00 +CPU= 18.296 14:52:38 STEP 202 H= 202:00 +CPU= 11.802 14:52:50 STEP 203 H= 203:00 +CPU= 11.310 14:53:10 STEP 204 H= 204:00 +CPU= 20.217 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000204 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000204 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000204 14:53:21 STEP 205 H= 205:00 +CPU= 11.011 14:53:33 STEP 206 H= 206:00 +CPU= 11.746 14:53:51 STEP 207 H= 207:00 +CPU= 17.888 14:54:02 STEP 208 H= 208:00 +CPU= 11.504 14:54:14 STEP 209 H= 209:00 +CPU= 11.490 14:54:33 STEP 210 H= 210:00 +CPU= 18.572 14:54:44 STEP 211 H= 211:00 +CPU= 11.258 14:54:56 STEP 212 H= 212:00 +CPU= 11.901 14:55:14 STEP 213 H= 213:00 +CPU= 18.215 14:55:25 STEP 214 H= 214:00 +CPU= 11.152 14:55:37 STEP 215 H= 215:00 +CPU= 11.557 14:56:27 STEP 216 H= 216:00 +CPU= 21.507 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000216 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000216 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000204 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000204 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000204 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096_7.zip Uploading the intermediate file: upload_file_7.zip 14:56:38 STEP 217 H= 217:00 +CPU= 11.004 14:56:50 STEP 218 H= 218:00 +CPU= 11.562 14:57:08 STEP 219 H= 219:00 +CPU= 18.160 14:57:19 STEP 220 H= 220:00 +CPU= 11.228 14:57:31 STEP 221 H= 221:00 +CPU= 11.042 14:57:50 STEP 222 H= 222:00 +CPU= 18.453 14:58:02 STEP 223 H= 223:00 +CPU= 11.067 14:58:13 STEP 224 H= 224:00 +CPU= 11.257 14:58:32 STEP 225 H= 225:00 +CPU= 17.941 14:58:44 STEP 226 H= 226:00 +CPU= 11.732 14:58:55 STEP 227 H= 227:00 +CPU= 11.064 14:59:15 STEP 228 H= 228:00 +CPU= 20.163 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000228 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000228 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000228 14:59:27 STEP 229 H= 229:00 +CPU= 11.363 14:59:38 STEP 230 H= 230:00 +CPU= 11.340 14:59:57 STEP 231 H= 231:00 +CPU= 18.511 15:00:08 STEP 232 H= 232:00 +CPU= 11.661 15:00:19 STEP 233 H= 233:00 +CPU= 10.765 15:00:37 STEP 234 H= 234:00 +CPU= 18.233 15:00:49 STEP 235 H= 235:00 +CPU= 11.363 15:01:02 STEP 236 H= 236:00 +CPU= 11.336 15:01:20 STEP 237 H= 237:00 +CPU= 17.747 15:01:33 STEP 238 H= 238:00 +CPU= 11.569 15:01:45 STEP 239 H= 239:00 +CPU= 11.345 15:02:46 STEP 240 H= 240:00 +CPU= 21.281 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000240 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000240 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMGGhq0f+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMSHhq0f+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869/ICMUAhq0f+000228 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096_8.zip Uploading the intermediate file: upload_file_8.zip 15:02:59 STEP 241 H= 241:00 +CPU= 10.992 15:03:11 STEP 242 H= 242:00 +CPU= 11.291 15:03:29 STEP 243 H= 243:00 +CPU= 17.877 15:03:40 STEP 244 H= 244:00 +CPU= 11.736 15:03:52 STEP 245 H= 245:00 +CPU= 11.212 15:04:10 STEP 246 H= 246:00 +CPU= 18.351 15:04:21 STEP 247 H= 247:00 +CPU= 11.379 15:04:33 STEP 248 H= 248:00 +CPU= 11.739 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0225 (argv4) batchid: 975 (argv5) wuid: 12191869 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_a07c2daf6194f196239a169d72880610 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip ic_ancil_file: ic_ancil_12191869 ifsdata_file: ifsdata_12191869 climate_data_file: clim_data_12191869 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3d219496834644728cf43598af0f78a to: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_368d528635a82655dd2b0132d4056598 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12191869.xml -- Model is restarting -- Adjusting last_iter, 249, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml last_cpu_time: 4841 upload_file_number: 9 last_iter: 239 last_upload: 864000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096 The child process has been launched with process id: 72952 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72952:72952] [20230102:150951:1672697391:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:10:44 STEP 240 H= 240:00 +CPU= 20.690 15:10:57 STEP 241 H= 241:00 +CPU= 11.434 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0225 (argv4) batchid: 975 (argv5) wuid: 12191869 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_a07c2daf6194f196239a169d72880610 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip ic_ancil_file: ic_ancil_12191869 ifsdata_file: ifsdata_12191869 climate_data_file: clim_data_12191869 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3d219496834644728cf43598af0f78a to: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_368d528635a82655dd2b0132d4056598 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12191869.xml -- Model is restarting -- Adjusting last_iter, 242, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml last_cpu_time: 4878 upload_file_number: 9 last_iter: 239 last_upload: 864000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096 The child process has been launched with process id: 72972 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:72972:72972] [20230102:151606:1672697766:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:16:57 STEP 240 H= 240:00 +CPU= 20.085 15:17:09 STEP 241 H= 241:00 +CPU= 11.393 15:17:20 STEP 242 H= 242:00 +CPU= 11.258 15:17:37 STEP 243 H= 243:00 +CPU= 17.295 15:17:49 STEP 244 H= 244:00 +CPU= 11.239 15:18:00 STEP 245 H= 245:00 +CPU= 11.063 15:18:18 STEP 246 H= 246:00 +CPU= 17.873 15:18:29 STEP 247 H= 247:00 +CPU= 11.026 15:18:40 STEP 248 H= 248:00 +CPU= 11.352 15:18:58 STEP 249 H= 249:00 +CPU= 17.909 15:19:10 STEP 250 H= 250:00 +CPU= 11.583 15:19:21 STEP 251 H= 251:00 +CPU= 11.244 15:19:42 STEP 252 H= 252:00 +CPU= 20.012 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+000252 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+000252 Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+000252 15:19:53 STEP 253 H= 253:00 +CPU= 11.004 15:20:05 STEP 254 H= 254:00 +CPU= 11.386 15:20:23 STEP 255 H= 255:00 +CPU= 18.009 15:20:34 STEP 256 H= 256:00 +CPU= 11.296 15:20:45 STEP 257 H= 257:00 +CPU= 10.991 15:21:04 STEP 258 H= 258:00 +CPU= 18.418 15:21:15 STEP 259 H= 259:00 +CPU= 11.069 15:21:26 STEP 260 H= 260:00 +CPU= 11.242 15:21:43 STEP 261 H= 261:00 +CPU= 17.432 15:21:55 STEP 262 H= 262:00 +CPU= 11.324 15:22:12 STEP 263 H= 263:00 +CPU= 11.604 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0225 (argv4) batchid: 975 (argv5) wuid: 12191869 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_a07c2daf6194f196239a169d72880610 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip ic_ancil_file: ic_ancil_12191869 ifsdata_file: ifsdata_12191869 climate_data_file: clim_data_12191869 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3d219496834644728cf43598af0f78a to: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_368d528635a82655dd2b0132d4056598 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12191869.xml -- Model is restarting -- Adjusting last_iter, 264, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml last_cpu_time: 5208 upload_file_number: 9 last_iter: 263 last_upload: 864000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096 The child process has been launched with process id: 73002 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73002:73002] [20230102:152515:1672698315:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 2006050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0225 (argv4) batchid: 975 (argv5) wuid: 12191869 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/3 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12191869 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_a07c2daf6194f196239a169d72880610 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0225_2006050100_123_975_12191869.zip ic_ancil_file: ic_ancil_12191869 ifsdata_file: ifsdata_12191869 climate_data_file: clim_data_12191869 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_b3d219496834644728cf43598af0f78a to: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12191869.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_368d528635a82655dd2b0132d4056598 to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12191869.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12191869.zip Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12191869.xml -- Model is restarting -- Adjusting last_iter, 240, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12191869.xml last_cpu_time: 5211 upload_file_number: 9 last_iter: 239 last_upload: 864000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0225_2006050100_123_975_12191869_0_r754105096 The child process has been launched with process id: 73012 Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152802:1672698482:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> ABORT! 1 !! *** WAVE MODEL HAS ABORTED [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152821:1672698501:19.494] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1 Process 1 thread 1 calling linux_trbk from intel_trbk() [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/3/oifs_43r3_model.exe' : sigcontextptr=0x7fffffbad220 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/3/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0 [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s) SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152822:1672698502:20.301] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152822:1672698502:20.301] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152822:1672698502:20.301] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil)) [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152822:1672698502:20.301] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x7100011d34], nsigs = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152822:1672698502:20.301] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152823:1672698503:21.302] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1 [EC_DRHOOK:3900x-2:1:1:73012:73012] [20230102:152823:1672698503:21.302] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1 forrtl: error (76): Abort trap signal Image PC Routine Line Source oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DC903B Unknown Unknown Unknown oifs_43r3_model.e 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90 oifs_43r3_model.e 000000000081D8AF abor1_ 41 abor1.F90 oifs_43r3_model.e 00000000015F8ACA mpabort_ 50 mpabort.F oifs_43r3_model.e 00000000015D51C8 abort1_ 44 abort1.F oifs_43r3_model.e 000000000166B07F userin_ 553 userin.F oifs_43r3_model.e 00000000015D7FF0 initmdl_ 452 initmdl.F oifs_43r3_model.e 00000000008711E0 wavemdl_ 393 wavemdl.F oifs_43r3_model.e 00000000007E0165 wvcouple_ 260 wvcouple.F90 oifs_43r3_model.e 000000000055E4AD cnt4_ 1283 cnt4.F90 oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90 oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown The child process terminated with status: 0 ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org