Name | oifs_43r3_ps_0993_1982050100_123_951_12168637_0 |
Workunit | 12168637 |
Created | 20 Dec 2022, 23:35:22 UTC |
Sent | 20 Dec 2022, 23:37:37 UTC |
Report deadline | 19 Jan 2023, 23:37:37 UTC |
Received | 23 Dec 2022, 22:02:07 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1467890 |
Run time | 6 hours 29 min 14 sec |
CPU time | 3 hours 52 min 23 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 4.20 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,408.51 MB |
Peak swap size | 4,978.07 MB |
Peak disk usage | 7,718.78 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> CALLSTACK=0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:989138:989138] [20221223:052739:1671769659:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 05:28:46 STEP 216 H= 216:00 +CPU= 55.103 05:29:16 STEP 217 H= 217:00 +CPU= 29.967 05:29:48 STEP 218 H= 218:00 +CPU= 31.779 05:30:37 STEP 219 H= 219:00 +CPU= 49.226 05:31:08 STEP 220 H= 220:00 +CPU= 31.024 05:31:39 STEP 221 H= 221:00 +CPU= 30.798 05:32:29 STEP 222 H= 222:00 +CPU= 49.902 05:32:59 STEP 223 H= 223:00 +CPU= 30.124 05:33:30 STEP 224 H= 224:00 +CPU= 31.100 05:34:19 STEP 225 H= 225:00 +CPU= 48.250 05:34:50 STEP 226 H= 226:00 +CPU= 30.780 05:35:21 STEP 227 H= 227:00 +CPU= 30.991 05:36:15 STEP 228 H= 228:00 +CPU= 53.971 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000228 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000228 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000228 05:36:45 STEP 229 H= 229:00 +CPU= 30.019 05:37:16 STEP 230 H= 230:00 +CPU= 31.041 05:38:04 STEP 231 H= 231:00 +CPU= 47.799 05:38:35 STEP 232 H= 232:00 +CPU= 30.554 05:39:05 STEP 233 H= 233:00 +CPU= 30.145 05:39:55 STEP 234 H= 234:00 +CPU= 49.279 05:40:26 STEP 235 H= 235:00 +CPU= 30.774 05:40:58 STEP 236 H= 236:00 +CPU= 32.413 05:41:48 STEP 237 H= 237:00 +CPU= 49.783 05:42:19 STEP 238 H= 238:00 +CPU= 30.888 05:42:52 STEP 239 H= 239:00 +CPU= 32.025 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000228 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151_7.zip Uploading the intermediate file: upload_file_7.zip 05:43:53 STEP 240 H= 240:00 +CPU= 58.245 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000240 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000240 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000240 05:44:23 STEP 241 H= 241:00 +CPU= 29.869 05:44:55 STEP 242 H= 242:00 +CPU= 32.056 05:45:44 STEP 243 H= 243:00 +CPU= 48.028 05:46:15 STEP 244 H= 244:00 +CPU= 30.814 05:46:46 STEP 245 H= 245:00 +CPU= 31.030 05:47:36 STEP 246 H= 246:00 +CPU= 49.692 05:48:06 STEP 247 H= 247:00 +CPU= 30.076 05:48:38 STEP 248 H= 248:00 +CPU= 31.476 05:49:26 STEP 249 H= 249:00 +CPU= 48.352 05:49:57 STEP 250 H= 250:00 +CPU= 30.729 05:50:28 STEP 251 H= 251:00 +CPU= 30.605 05:51:21 STEP 252 H= 252:00 +CPU= 52.816 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000252 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000252 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000252 05:51:51 STEP 253 H= 253:00 +CPU= 29.940 05:52:22 STEP 254 H= 254:00 +CPU= 31.070 05:53:10 STEP 255 H= 255:00 +CPU= 48.154 05:53:41 STEP 256 H= 256:00 +CPU= 31.063 05:54:11 STEP 257 H= 257:00 +CPU= 29.826 05:55:01 STEP 258 H= 258:00 +CPU= 49.985 05:55:31 STEP 259 H= 259:00 +CPU= 29.586 05:56:06 STEP 260 H= 260:00 +CPU= 33.763 05:56:53 STEP 261 H= 261:00 +CPU= 47.427 05:57:24 STEP 262 H= 262:00 +CPU= 30.810 05:57:55 STEP 263 H= 263:00 +CPU= 30.638 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000252 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000252 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000252 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151_8.zip Uploading the intermediate file: upload_file_8.zip 05:58:51 STEP 264 H= 264:00 +CPU= 55.768 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000264 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000264 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000264 05:59:21 STEP 265 H= 265:00 +CPU= 30.303 05:59:49 STEP 266 H= 266:00 +CPU= 27.002 06:00:36 STEP 267 H= 267:00 +CPU= 46.935 06:01:06 STEP 268 H= 268:00 +CPU= 30.321 06:01:38 STEP 269 H= 269:00 +CPU= 31.464 06:02:28 STEP 270 H= 270:00 +CPU= 49.885 06:02:58 STEP 271 H= 271:00 +CPU= 30.575 06:03:31 STEP 272 H= 272:00 +CPU= 32.220 06:04:19 STEP 273 H= 273:00 +CPU= 47.991 06:04:50 STEP 274 H= 274:00 +CPU= 31.254 06:05:21 STEP 275 H= 275:00 +CPU= 30.758 06:06:14 STEP 276 H= 276:00 +CPU= 53.303 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000276 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000276 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000276 06:06:45 STEP 277 H= 277:00 +CPU= 30.151 06:07:16 STEP 278 H= 278:00 +CPU= 31.293 06:08:05 STEP 279 H= 279:00 +CPU= 49.063 06:08:36 STEP 280 H= 280:00 +CPU= 31.157 06:09:06 STEP 281 H= 281:00 +CPU= 29.681 06:09:57 STEP 282 H= 282:00 +CPU= 50.215 06:10:27 STEP 283 H= 283:00 +CPU= 29.871 06:10:59 STEP 284 H= 284:00 +CPU= 32.731 06:11:47 STEP 285 H= 285:00 +CPU= 47.394 06:12:18 STEP 286 H= 286:00 +CPU= 31.256 06:12:49 STEP 287 H= 287:00 +CPU= 30.465 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000264 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000264 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000264 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000276 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000276 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000276 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151_9.zip Uploading the intermediate file: upload_file_9.zip 06:13:50 STEP 288 H= 288:00 +CPU= 60.755 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000288 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000288 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000288 06:14:36 STEP 289 H= 289:00 +CPU= 31.314 06:15:10 STEP 290 H= 290:00 +CPU= 32.614 06:16:00 STEP 291 H= 291:00 +CPU= 47.609 06:16:31 STEP 292 H= 292:00 +CPU= 30.823 06:17:01 STEP 293 H= 293:00 +CPU= 29.974 06:17:54 STEP 294 H= 294:00 +CPU= 52.293 06:18:24 STEP 295 H= 295:00 +CPU= 30.179 06:18:57 STEP 296 H= 296:00 +CPU= 32.660 06:19:48 STEP 297 H= 297:00 +CPU= 50.080 06:20:20 STEP 298 H= 298:00 +CPU= 32.710 06:20:52 STEP 299 H= 299:00 +CPU= 31.470 06:21:52 STEP 300 H= 300:00 +CPU= 57.135 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000300 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000300 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000300 06:22:23 STEP 301 H= 301:00 +CPU= 30.691 06:22:55 STEP 302 H= 302:00 +CPU= 31.333 06:23:46 STEP 303 H= 303:00 +CPU= 51.257 06:24:18 STEP 304 H= 304:00 +CPU= 31.586 06:24:51 STEP 305 H= 305:00 +CPU= 32.796 06:25:43 STEP 306 H= 306:00 +CPU= 51.331 06:26:15 STEP 307 H= 307:00 +CPU= 32.222 06:26:48 STEP 308 H= 308:00 +CPU= 32.987 06:27:38 STEP 309 H= 309:00 +CPU= 49.774 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1982050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0993 (argv4) batchid: 951 (argv5) wuid: 12168637 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/5 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_8146efca7c5f89696fda72308079e31c to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip ic_ancil_file: ic_ancil_12168637 ifsdata_file: ifsdata_12168637 climate_data_file: clim_data_12168637 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_ae2e28a63c35a79c1304fbc72a2dcf85 to: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_972cab665464e91b7cee86f05d06e04e to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12168637.xml -- Model is restarting -- Adjusting last_iter, 310, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml last_cpu_time: 10528 upload_file_number: 10 last_iter: 287 last_upload: 1033200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151 The child process has been launched with process id: 996137 Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:996137:996137] [20221223:062900:1671773340:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 06:30:33 STEP 288 H= 288:00 +CPU= 55.533 06:31:06 STEP 289 H= 289:00 +CPU= 32.292 06:31:38 STEP 290 H= 290:00 +CPU= 32.173 06:32:28 STEP 291 H= 291:00 +CPU= 49.466 06:33:00 STEP 292 H= 292:00 +CPU= 31.897 06:33:32 STEP 293 H= 293:00 +CPU= 31.911 06:34:23 STEP 294 H= 294:00 +CPU= 51.412 06:34:55 STEP 295 H= 295:00 +CPU= 31.445 06:35:27 STEP 296 H= 296:00 +CPU= 31.949 06:36:19 STEP 297 H= 297:00 +CPU= 51.747 06:36:51 STEP 298 H= 298:00 +CPU= 31.399 06:37:23 STEP 299 H= 299:00 +CPU= 32.166 06:38:20 STEP 300 H= 300:00 +CPU= 55.639 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000300 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000300 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000300 06:38:52 STEP 301 H= 301:00 +CPU= 32.311 06:39:25 STEP 302 H= 302:00 +CPU= 33.099 06:40:16 STEP 303 H= 303:00 +CPU= 50.186 06:40:47 STEP 304 H= 304:00 +CPU= 31.435 06:41:19 STEP 305 H= 305:00 +CPU= 31.585 06:42:11 STEP 306 H= 306:00 +CPU= 51.796 06:42:42 STEP 307 H= 307:00 +CPU= 31.044 06:43:17 STEP 308 H= 308:00 +CPU= 33.526 06:44:07 STEP 309 H= 309:00 +CPU= 50.257 06:44:40 STEP 310 H= 310:00 +CPU= 32.661 06:45:12 STEP 311 H= 311:00 +CPU= 31.687 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000288 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000288 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000288 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000300 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000300 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000300 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151_10.zip Uploading the intermediate file: upload_file_10.zip 06:46:13 STEP 312 H= 312:00 +CPU= 59.219 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000312 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000312 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000312 06:46:44 STEP 313 H= 313:00 +CPU= 31.018 06:47:17 STEP 314 H= 314:00 +CPU= 32.926 06:48:07 STEP 315 H= 315:00 +CPU= 50.143 06:48:39 STEP 316 H= 316:00 +CPU= 31.616 06:49:09 STEP 317 H= 317:00 +CPU= 30.243 06:50:01 STEP 318 H= 318:00 +CPU= 51.628 06:50:32 STEP 319 H= 319:00 +CPU= 31.008 06:51:05 STEP 320 H= 320:00 +CPU= 33.146 06:51:55 STEP 321 H= 321:00 +CPU= 49.652 06:52:29 STEP 322 H= 322:00 +CPU= 33.994 06:53:01 STEP 323 H= 323:00 +CPU= 31.675 06:53:56 STEP 324 H= 324:00 +CPU= 54.857 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000324 06:54:28 STEP 325 H= 325:00 +CPU= 31.441 06:54:59 STEP 326 H= 326:00 +CPU= 30.897 06:55:50 STEP 327 H= 327:00 +CPU= 50.885 06:56:21 STEP 328 H= 328:00 +CPU= 30.858 06:56:55 STEP 329 H= 329:00 +CPU= 33.409 06:57:45 STEP 330 H= 330:00 +CPU= 49.388 06:58:18 STEP 331 H= 331:00 +CPU= 32.069 06:58:52 STEP 332 H= 332:00 +CPU= 33.143 06:59:41 STEP 333 H= 333:00 +CPU= 48.909 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1982050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0993 (argv4) batchid: 951 (argv5) wuid: 12168637 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/5 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_8146efca7c5f89696fda72308079e31c to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip ic_ancil_file: ic_ancil_12168637 ifsdata_file: ifsdata_12168637 climate_data_file: clim_data_12168637 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_ae2e28a63c35a79c1304fbc72a2dcf85 to: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_972cab665464e91b7cee86f05d06e04e to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12168637.xml -- Model is restarting -- Adjusting last_iter, 334, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml last_cpu_time: 12336 upload_file_number: 11 last_iter: 311 last_upload: 1119600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151 The child process has been launched with process id: 1053821 Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1053821:1053821] [20221223:213047:1671827447:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 21:39:09 STEP 312 H= 312:00 +CPU= 47.955 21:41:42 STEP 313 H= 313:00 +CPU= 21.323 21:43:59 STEP 314 H= 314:00 +CPU= 22.187 21:45:18 STEP 315 H= 315:00 +CPU= 31.966 21:46:07 STEP 316 H= 316:00 +CPU= 21.245 21:46:31 STEP 317 H= 317:00 +CPU= 20.546 21:47:26 STEP 318 H= 318:00 +CPU= 34.974 21:47:52 STEP 319 H= 319:00 +CPU= 20.779 21:48:38 STEP 320 H= 320:00 +CPU= 21.171 21:50:01 STEP 321 H= 321:00 +CPU= 33.557 21:50:28 STEP 322 H= 322:00 +CPU= 22.996 21:50:49 STEP 323 H= 323:00 +CPU= 20.328 21:52:16 STEP 324 H= 324:00 +CPU= 38.656 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000324 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000324 21:52:39 STEP 325 H= 325:00 +CPU= 22.876 21:53:06 STEP 326 H= 326:00 +CPU= 26.600 21:53:44 STEP 327 H= 327:00 +CPU= 36.880 21:54:13 STEP 328 H= 328:00 +CPU= 23.075 21:54:40 STEP 329 H= 329:00 +CPU= 22.784 21:55:39 STEP 330 H= 330:00 +CPU= 35.792 21:56:20 STEP 331 H= 331:00 +CPU= 20.913 21:57:00 STEP 332 H= 332:00 +CPU= 33.251 21:57:48 STEP 333 H= 333:00 +CPU= 36.108 21:58:13 STEP 334 H= 334:00 +CPU= 25.132 21:58:55 STEP 335 H= 335:00 +CPU= 25.328 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000312 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000312 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000312 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000324 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000324 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000324 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151_11.zip Uploading the intermediate file: upload_file_11.zip 22:01:16 STEP 336 H= 336:00 +CPU= 37.492 Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1982050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0993 (argv4) batchid: 951 (argv5) wuid: 12168637 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/5 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_8146efca7c5f89696fda72308079e31c to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip ic_ancil_file: ic_ancil_12168637 ifsdata_file: ifsdata_12168637 climate_data_file: clim_data_12168637 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_ae2e28a63c35a79c1304fbc72a2dcf85 to: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_972cab665464e91b7cee86f05d06e04e to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12168637.xml -- Model is restarting -- Adjusting last_iter, 336, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml last_cpu_time: 13039 upload_file_number: 12 last_iter: 335 last_upload: 1206000 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151 The child process has been launched with process id: 1054859 Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1054859:1054859] [20221223:220311:1671829391:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 22:04:35 STEP 336 H= 336:00 +CPU= 42.477 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000336 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000336 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000336 22:05:01 STEP 337 H= 337:00 +CPU= 25.032 22:05:27 STEP 338 H= 338:00 +CPU= 25.756 22:06:07 STEP 339 H= 339:00 +CPU= 40.632 22:06:34 STEP 340 H= 340:00 +CPU= 26.237 22:06:57 STEP 341 H= 341:00 +CPU= 23.508 22:07:47 STEP 342 H= 342:00 +CPU= 49.513 22:08:13 STEP 343 H= 343:00 +CPU= 26.388 22:08:45 STEP 344 H= 344:00 +CPU= 31.299 22:09:29 STEP 345 H= 345:00 +CPU= 43.862 22:10:00 STEP 346 H= 346:00 +CPU= 31.525 22:10:31 STEP 347 H= 347:00 +CPU= 30.358 22:11:21 STEP 348 H= 348:00 +CPU= 49.886 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000348 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000348 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000348 22:11:48 STEP 349 H= 349:00 +CPU= 27.285 22:12:19 STEP 350 H= 350:00 +CPU= 30.393 22:13:08 STEP 351 H= 351:00 +CPU= 48.810 22:13:37 STEP 352 H= 352:00 +CPU= 29.111 22:14:02 STEP 353 H= 353:00 +CPU= 24.422 22:14:49 STEP 354 H= 354:00 +CPU= 47.895 22:15:13 STEP 355 H= 355:00 +CPU= 23.727 22:15:42 STEP 356 H= 356:00 +CPU= 28.252 22:16:31 STEP 357 H= 357:00 +CPU= 49.489 22:16:57 STEP 358 H= 358:00 +CPU= 25.635 22:17:31 STEP 359 H= 359:00 +CPU= 33.625 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000336 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000336 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000336 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMGGhq0f+000348 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMSHhq0f+000348 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637/ICMUAhq0f+000348 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151_12.zip Uploading the intermediate file: upload_file_12.zip 22:18:32 STEP 360 H= 360:00 +CPU= 58.121 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+000360 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+000360 Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+000360 22:19:00 STEP 361 H= 361:00 +CPU= 28.137 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1982050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0993 (argv4) batchid: 951 (argv5) wuid: 12168637 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/5 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12168637 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_8146efca7c5f89696fda72308079e31c to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0993_1982050100_123_951_12168637.zip ic_ancil_file: ic_ancil_12168637 ifsdata_file: ifsdata_12168637 climate_data_file: clim_data_12168637 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_ae2e28a63c35a79c1304fbc72a2dcf85 to: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12168637.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_972cab665464e91b7cee86f05d06e04e to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12168637.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12168637.zip Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12168637.xml -- Model is restarting -- Adjusting last_iter, 362, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12168637.xml last_cpu_time: 13943 upload_file_number: 13 last_iter: 359 last_upload: 1292400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0993_1982050100_123_951_12168637_0_r1482334151 The child process has been launched with process id: 1056018 Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:thomas-HP-ENVY-dv7-Notebook-PC:1:1:1056018:1056018] [20221223:222042:1671830442:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org