Name | oifs_43r3_ps_0089_1989050100_123_958_12174733_0 |
Workunit | 12174733 |
Created | 21 Dec 2022, 15:38:13 UTC |
Sent | 21 Dec 2022, 15:42:10 UTC |
Report deadline | 20 Jan 2023, 15:42:10 UTC |
Received | 22 Dec 2022, 9:37:16 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1519251 |
Run time | 5 hours 53 min 5 sec |
CPU time | 5 hours 44 min 29 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 4.97 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,415.49 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,682.02 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> ) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3209560:3209560] [20221222:110403:1671674643:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:06:06 STEP 504 H= 504:00 +CPU= 42.294 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:11:09 STEP 505 H= 505:00 +CPU= 22.643 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:13:53 STEP 506 H= 506:00 +CPU= 22.248 11:14:27 STEP 507 H= 507:00 +CPU= 34.219 11:14:49 STEP 508 H= 508:00 +CPU= 21.904 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:22:52 STEP 509 H= 509:00 +CPU= 20.354 11:23:30 STEP 510 H= 510:00 +CPU= 37.213 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:27:05 STEP 511 H= 511:00 +CPU= 23.223 11:27:31 STEP 512 H= 512:00 +CPU= 24.876 11:28:08 STEP 513 H= 513:00 +CPU= 36.289 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:44:55 STEP 514 H= 514:00 +CPU= 24.312 11:45:20 STEP 515 H= 515:00 +CPU= 24.439 11:46:02 STEP 516 H= 516:00 +CPU= 41.873 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000516 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000516 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000516 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:47:44 STEP 517 H= 517:00 +CPU= 20.942 11:48:05 STEP 518 H= 518:00 +CPU= 21.057 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:49:50 STEP 519 H= 519:00 +CPU= 32.870 11:50:11 STEP 520 H= 520:00 +CPU= 21.306 11:50:32 STEP 521 H= 521:00 +CPU= 20.515 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:52:16 STEP 522 H= 522:00 +CPU= 33.491 11:52:36 STEP 523 H= 523:00 +CPU= 20.391 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:54:29 STEP 524 H= 524:00 +CPU= 20.854 11:55:03 STEP 525 H= 525:00 +CPU= 33.242 11:55:23 STEP 526 H= 526:00 +CPU= 20.739 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:56:36 STEP 527 H= 527:00 +CPU= 21.422 11:57:17 STEP 528 H= 528:00 +CPU= 37.719 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000528 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000528 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000504 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000504 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000504 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000516 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000516 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000516 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_21.zip Uploading the intermediate file: upload_file_21.zip 11:57:36 STEP 529 H= 529:00 +CPU= 19.006 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:59:03 STEP 530 H= 530:00 +CPU= 21.320 11:59:35 STEP 531 H= 531:00 +CPU= 32.079 11:59:56 STEP 532 H= 532:00 +CPU= 20.751 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:01:27 STEP 533 H= 533:00 +CPU= 20.784 12:01:59 STEP 534 H= 534:00 +CPU= 32.471 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:03:31 STEP 535 H= 535:00 +CPU= 20.263 12:03:53 STEP 536 H= 536:00 +CPU= 22.113 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:05:37 STEP 537 H= 537:00 +CPU= 32.450 12:05:58 STEP 538 H= 538:00 +CPU= 21.095 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:07:19 STEP 539 H= 539:00 +CPU= 20.937 12:07:55 STEP 540 H= 540:00 +CPU= 35.547 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000540 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000540 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000540 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:09:16 STEP 541 H= 541:00 +CPU= 20.204 12:09:38 STEP 542 H= 542:00 +CPU= 22.069 12:10:10 STEP 543 H= 543:00 +CPU= 31.533 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:11:54 STEP 544 H= 544:00 +CPU= 21.573 12:12:15 STEP 545 H= 545:00 +CPU= 20.616 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:13:49 STEP 546 H= 546:00 +CPU= 33.968 12:14:09 STEP 547 H= 547:00 +CPU= 20.407 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:15:43 STEP 548 H= 548:00 +CPU= 21.374 12:16:15 STEP 549 H= 549:00 +CPU= 32.458 12:16:37 STEP 550 H= 550:00 +CPU= 21.019 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:18:18 STEP 551 H= 551:00 +CPU= 21.557 12:18:57 STEP 552 H= 552:00 +CPU= 38.184 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000552 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000552 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000540 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000540 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000540 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_22.zip Uploading the intermediate file: upload_file_22.zip 12:19:16 STEP 553 H= 553:00 +CPU= 19.366 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:21:00 STEP 554 H= 554:00 +CPU= 20.736 12:21:33 STEP 555 H= 555:00 +CPU= 32.398 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:23:35 STEP 556 H= 556:00 +CPU= 21.065 12:23:57 STEP 557 H= 557:00 +CPU= 21.197 12:24:30 STEP 558 H= 558:00 +CPU= 33.381 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:26:22 STEP 559 H= 559:00 +CPU= 20.159 12:26:43 STEP 560 H= 560:00 +CPU= 21.254 12:27:19 STEP 561 H= 561:00 +CPU= 35.159 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:33:05 STEP 562 H= 562:00 +CPU= 24.678 12:33:29 STEP 563 H= 563:00 +CPU= 23.428 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:52:13 STEP 564 H= 564:00 +CPU= 40.271 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000564 12:52:36 STEP 565 H= 565:00 +CPU= 22.405 12:53:02 STEP 566 H= 566:00 +CPU= 25.283 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:00:43 STEP 567 H= 567:00 +CPU= 37.313 13:01:07 STEP 568 H= 568:00 +CPU= 23.448 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0089 (argv4) batchid: 958 (argv5) wuid: 12174733 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/6 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3b888d78cb29e0b788044e738156bcdf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip ic_ancil_file: ic_ancil_12174733 ifsdata_file: ifsdata_12174733 climate_data_file: clim_data_12174733 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_41198eeba27a163b4104a5ac47617315 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12174733.xml -- Model is restarting -- Adjusting last_iter, 569, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml last_cpu_time: 16099 upload_file_number: 23 last_iter: 551 last_upload: 1987200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189 The child process has been launched with process id: 3234652 Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234652:3234652] [20221222:132037:1671682837:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:25:28 STEP 552 H= 552:00 +CPU= 40.885 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:34:30 STEP 553 H= 553:00 +CPU= 21.178 13:34:52 STEP 554 H= 554:00 +CPU= 21.808 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:36:58 STEP 555 H= 555:00 +CPU= 33.258 13:37:19 STEP 556 H= 556:00 +CPU= 21.058 13:37:40 STEP 557 H= 557:00 +CPU= 20.549 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:39:35 STEP 558 H= 558:00 +CPU= 34.194 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:41:07 STEP 559 H= 559:00 +CPU= 20.833 13:41:29 STEP 560 H= 560:00 +CPU= 21.121 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:43:23 STEP 561 H= 561:00 +CPU= 32.833 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:44:55 STEP 562 H= 562:00 +CPU= 20.557 13:45:17 STEP 563 H= 563:00 +CPU= 21.729 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:47:03 STEP 564 H= 564:00 +CPU= 35.814 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000564 13:48:36 STEP 565 H= 565:00 +CPU= 21.125 13:48:57 STEP 566 H= 566:00 +CPU= 21.029 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:50:51 STEP 567 H= 567:00 +CPU= 33.116 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:52:14 STEP 568 H= 568:00 +CPU= 21.678 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:53:26 STEP 569 H= 569:00 +CPU= 20.977 13:53:59 STEP 570 H= 570:00 +CPU= 32.912 13:54:20 STEP 571 H= 571:00 +CPU= 20.615 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:56:01 STEP 572 H= 572:00 +CPU= 21.168 13:56:34 STEP 573 H= 573:00 +CPU= 32.089 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:58:36 STEP 574 H= 574:00 +CPU= 21.266 13:58:58 STEP 575 H= 575:00 +CPU= 21.486 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:00:48 STEP 576 H= 576:00 +CPU= 38.979 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000576 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000576 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000564 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000564 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000564 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_23.zip Uploading the intermediate file: upload_file_23.zip 14:01:09 STEP 577 H= 577:00 +CPU= 20.437 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:03:03 STEP 578 H= 578:00 +CPU= 21.842 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:04:38 STEP 579 H= 579:00 +CPU= 33.009 14:04:59 STEP 580 H= 580:00 +CPU= 21.134 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:06:51 STEP 581 H= 581:00 +CPU= 20.301 14:07:25 STEP 582 H= 582:00 +CPU= 34.187 14:07:45 STEP 583 H= 583:00 +CPU= 20.347 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:10:20 STEP 584 H= 584:00 +CPU= 21.874 14:10:52 STEP 585 H= 585:00 +CPU= 32.419 14:11:13 STEP 586 H= 586:00 +CPU= 20.905 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:12:55 STEP 587 H= 587:00 +CPU= 21.014 14:13:31 STEP 588 H= 588:00 +CPU= 35.537 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000588 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000588 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000588 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:15:24 STEP 589 H= 589:00 +CPU= 21.137 14:15:45 STEP 590 H= 590:00 +CPU= 21.242 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:17:49 STEP 591 H= 591:00 +CPU= 32.838 14:18:10 STEP 592 H= 592:00 +CPU= 21.078 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:19:52 STEP 593 H= 593:00 +CPU= 20.347 14:20:25 STEP 594 H= 594:00 +CPU= 33.655 14:20:45 STEP 595 H= 595:00 +CPU= 19.786 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:22:37 STEP 596 H= 596:00 +CPU= 21.851 14:23:10 STEP 597 H= 597:00 +CPU= 32.497 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:25:23 STEP 598 H= 598:00 +CPU= 20.836 14:25:45 STEP 599 H= 599:00 +CPU= 21.442 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:27:46 STEP 600 H= 600:00 +CPU= 39.316 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000600 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000600 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000588 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000588 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000588 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_24.zip Uploading the intermediate file: upload_file_24.zip 14:28:06 STEP 601 H= 601:00 +CPU= 20.063 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:30:06 STEP 602 H= 602:00 +CPU= 21.329 14:30:39 STEP 603 H= 603:00 +CPU= 32.989 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:32:51 STEP 604 H= 604:00 +CPU= 21.344 14:33:13 STEP 605 H= 605:00 +CPU= 21.186 14:33:46 STEP 606 H= 606:00 +CPU= 33.151 14:34:06 STEP 607 H= 607:00 +CPU= 19.999 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:35:49 STEP 608 H= 608:00 +CPU= 21.747 14:36:22 STEP 609 H= 609:00 +CPU= 32.559 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:37:57 STEP 610 H= 610:00 +CPU= 22.606 14:38:18 STEP 611 H= 611:00 +CPU= 20.889 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:40:07 STEP 612 H= 612:00 +CPU= 36.635 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000612 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000612 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000612 14:42:19 STEP 613 H= 613:00 +CPU= 21.036 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:44:43 STEP 614 H= 614:00 +CPU= 21.616 14:45:17 STEP 615 H= 615:00 +CPU= 33.212 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:48:09 STEP 616 H= 616:00 +CPU= 21.845 14:48:32 STEP 617 H= 617:00 +CPU= 22.455 14:49:06 STEP 618 H= 618:00 +CPU= 33.591 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:50:48 STEP 619 H= 619:00 +CPU= 21.525 14:51:10 STEP 620 H= 620:00 +CPU= 21.091 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:53:04 STEP 621 H= 621:00 +CPU= 33.409 14:53:25 STEP 622 H= 622:00 +CPU= 20.755 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:55:08 STEP 623 H= 623:00 +CPU= 21.705 14:55:47 STEP 624 H= 624:00 +CPU= 38.405 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000624 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000624 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000612 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000612 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000612 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_25.zip Uploading the intermediate file: upload_file_25.zip 14:58:01 STEP 625 H= 625:00 +CPU= 21.355 14:58:23 STEP 626 H= 626:00 +CPU= 22.114 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:00:18 STEP 627 H= 627:00 +CPU= 32.606 15:00:40 STEP 628 H= 628:00 +CPU= 21.442 15:01:00 STEP 629 H= 629:00 +CPU= 20.579 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:03:48 STEP 630 H= 630:00 +CPU= 33.416 15:04:09 STEP 631 H= 631:00 +CPU= 20.284 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:05:31 STEP 632 H= 632:00 +CPU= 21.294 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:06:44 STEP 633 H= 633:00 +CPU= 31.659 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:08:26 STEP 634 H= 634:00 +CPU= 21.337 15:08:47 STEP 635 H= 635:00 +CPU= 20.769 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:10:45 STEP 636 H= 636:00 +CPU= 36.234 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000636 15:11:06 STEP 637 H= 637:00 +CPU= 21.464 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:12:49 STEP 638 H= 638:00 +CPU= 21.409 15:13:21 STEP 639 H= 639:00 +CPU= 31.945 15:13:42 STEP 640 H= 640:00 +CPU= 20.957 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:15:34 STEP 641 H= 641:00 +CPU= 20.737 15:16:07 STEP 642 H= 642:00 +CPU= 33.250 15:16:27 STEP 643 H= 643:00 +CPU= 19.831 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:18:10 STEP 644 H= 644:00 +CPU= 21.417 15:18:42 STEP 645 H= 645:00 +CPU= 32.156 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:20:24 STEP 646 H= 646:00 +CPU= 21.322 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0089 (argv4) batchid: 958 (argv5) wuid: 12174733 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/6 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3b888d78cb29e0b788044e738156bcdf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip ic_ancil_file: ic_ancil_12174733 ifsdata_file: ifsdata_12174733 climate_data_file: clim_data_12174733 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_41198eeba27a163b4104a5ac47617315 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12174733.xml -- Model is restarting -- Adjusting last_iter, 647, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml last_cpu_time: 18531 upload_file_number: 26 last_iter: 623 last_upload: 2246400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189 The child process has been launched with process id: 3254598 Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254598:3254598] [20221222:152218:1671690138:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:23:00 STEP 624 H= 624:00 +CPU= 37.158 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:24:52 STEP 625 H= 625:00 +CPU= 20.875 15:25:13 STEP 626 H= 626:00 +CPU= 21.054 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:27:19 STEP 627 H= 627:00 +CPU= 32.884 15:27:40 STEP 628 H= 628:00 +CPU= 21.127 15:28:00 STEP 629 H= 629:00 +CPU= 19.928 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:30:04 STEP 630 H= 630:00 +CPU= 33.197 15:30:24 STEP 631 H= 631:00 +CPU= 19.818 15:30:45 STEP 632 H= 632:00 +CPU= 20.937 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:59 STEP 633 H= 633:00 +CPU= 32.890 15:33:20 STEP 634 H= 634:00 +CPU= 20.925 15:33:41 STEP 635 H= 635:00 +CPU= 20.305 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:48 STEP 636 H= 636:00 +CPU= 35.453 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000636 15:36:08 STEP 637 H= 637:00 +CPU= 19.985 15:36:29 STEP 638 H= 638:00 +CPU= 20.698 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:33 STEP 639 H= 639:00 +CPU= 33.256 15:38:53 STEP 640 H= 640:00 +CPU= 20.437 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:40:36 STEP 641 H= 641:00 +CPU= 21.338 15:41:09 STEP 642 H= 642:00 +CPU= 33.192 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:43:20 STEP 643 H= 643:00 +CPU= 20.218 15:43:42 STEP 644 H= 644:00 +CPU= 21.768 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:45:16 STEP 645 H= 645:00 +CPU= 32.636 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:46:40 STEP 646 H= 646:00 +CPU= 21.231 15:47:01 STEP 647 H= 647:00 +CPU= 21.685 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:49:01 STEP 648 H= 648:00 +CPU= 38.011 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000648 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000648 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000636 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000636 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000636 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_26.zip Uploading the intermediate file: upload_file_26.zip 15:49:21 STEP 649 H= 649:00 +CPU= 19.933 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:50:57 STEP 650 H= 650:00 +CPU= 21.510 15:51:30 STEP 651 H= 651:00 +CPU= 32.392 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:53:12 STEP 652 H= 652:00 +CPU= 21.267 15:53:34 STEP 653 H= 653:00 +CPU= 21.173 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:21 STEP 654 H= 654:00 +CPU= 34.388 16:01:43 STEP 655 H= 655:00 +CPU= 21.727 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:12:38 STEP 656 H= 656:00 +CPU= 22.479 16:13:13 STEP 657 H= 657:00 +CPU= 34.661 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:27 STEP 658 H= 658:00 +CPU= 22.305 16:17:51 STEP 659 H= 659:00 +CPU= 23.670 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:53 STEP 660 H= 660:00 +CPU= 39.110 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000660 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000660 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000660 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:28:57 STEP 661 H= 661:00 +CPU= 22.216 16:29:19 STEP 662 H= 662:00 +CPU= 22.529 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:36:55 STEP 663 H= 663:00 +CPU= 34.034 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:42:48 STEP 664 H= 664:00 +CPU= 21.836 16:43:10 STEP 665 H= 665:00 +CPU= 21.722 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:47:27 STEP 666 H= 666:00 +CPU= 35.396 16:47:50 STEP 667 H= 667:00 +CPU= 21.887 16:48:12 STEP 668 H= 668:00 +CPU= 22.324 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:55:00 STEP 669 H= 669:00 +CPU= 34.774 16:55:22 STEP 670 H= 670:00 +CPU= 22.478 16:55:45 STEP 671 H= 671:00 +CPU= 22.309 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:05:59 STEP 672 H= 672:00 +CPU= 40.911 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMGGhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMSHhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733/ICMUAhq0f+000660 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189_27.zip Uploading the intermediate file: upload_file_27.zip 17:06:21 STEP 673 H= 673:00 +CPU= 21.971 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:46 STEP 674 H= 674:00 +CPU= 22.679 17:13:22 STEP 675 H= 675:00 +CPU= 35.381 17:13:45 STEP 676 H= 676:00 +CPU= 22.456 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:19 STEP 677 H= 677:00 +CPU= 22.980 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:23:36 STEP 678 H= 678:00 +CPU= 35.501 Quit request received from BOINC client, ending the child process ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0089 (argv4) batchid: 958 (argv5) wuid: 12174733 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/6 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174733 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_3b888d78cb29e0b788044e738156bcdf to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0089_1989050100_123_958_12174733.zip ic_ancil_file: ic_ancil_12174733 ifsdata_file: ifsdata_12174733 climate_data_file: clim_data_12174733 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12174733.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_41198eeba27a163b4104a5ac47617315 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12174733.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12174733.zip Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12174733.xml -- Model is restarting -- Adjusting last_iter, 678, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12174733.xml last_cpu_time: 19971 upload_file_number: 28 last_iter: 671 last_upload: 2419200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0089_1989050100_123_958_12174733_0_r1233023189 The child process has been launched with process id: 3273887 Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3273887:3273887] [20221222:172922:1671697762:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:33:09 STEP 672 H= 672:00 +CPU= 40.062 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:36:04 STEP 673 H= 673:00 +CPU= 23.211 17:36:27 STEP 674 H= 674:00 +CPU= 22.815 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:40:45 STEP 675 H= 675:00 +CPU= 35.538 17:41:09 STEP 676 H= 676:00 +CPU= 23.264 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:47:13 STEP 677 H= 677:00 +CPU= 22.401 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:51:23 STEP 678 H= 678:00 +CPU= 36.801 17:51:46 STEP 679 H= 679:00 +CPU= 22.950 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:56:00 STEP 680 H= 680:00 +CPU= 22.684 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:58:58 STEP 681 H= 681:00 +CPU= 35.467 17:59:21 STEP 682 H= 682:00 +CPU= 22.874 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:02:45 STEP 683 H= 683:00 +CPU= 22.788 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:07:27 STEP 684 H= 684:00 +CPU= 38.569 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000684 18:07:51 STEP 685 H= 685:00 +CPU= 23.959 18:08:14 STEP 686 H= 686:00 +CPU= 22.537 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:11:51 STEP 687 H= 687:00 +CPU= 35.341 18:12:13 STEP 688 H= 688:00 +CPU= 22.312 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:18:07 STEP 689 H= 689:00 +CPU= 22.088 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:21:44 STEP 690 H= 690:00 +CPU= 35.871 18:22:08 STEP 691 H= 691:00 +CPU= 23.094 18:22:31 STEP 692 H= 692:00 +CPU= 22.536 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:25:49 STEP 693 H= 693:00 +CPU= 35.582 18:26:12 STEP 694 H= 694:00 +CPU= 23.049 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:30:18 STEP 695 H= 695:00 +CPU= 23.479 Abort request received from BOINC client, ending the child process ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 climateprediction.net