Name | oifs_43r3_ps_0096_1989050100_123_958_12174740_0 |
Workunit | 12174740 |
Created | 21 Dec 2022, 15:39:23 UTC |
Sent | 21 Dec 2022, 15:42:10 UTC |
Report deadline | 20 Jan 2023, 15:42:10 UTC |
Received | 22 Dec 2022, 11:17:29 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED |
Computer ID | 1519251 |
Run time | 5 hours 57 min 46 sec |
CPU time | 5 hours 50 min 6 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 4.97 GFLOPS |
Application version | OpenIFS 43r3 Perturbed Surface v1.05 x86_64-pc-linux-gnu |
Peak working set size | 4,410.86 MB |
Peak swap size | 4,978.08 MB |
Peak disk usage | 7,674.17 MB |
Stderr | <core_client_version>7.16.6</core_client_version> <![CDATA[ <message> Disk usage limit exceeded</message> <stderr_txt> the BOINC client, suspending the child process Resuming the child process 11:11:11 STEP 504 H= 504:00 +CPU= 42.145 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:13:53 STEP 505 H= 505:00 +CPU= 20.995 11:14:17 STEP 506 H= 506:00 +CPU= 23.595 11:14:51 STEP 507 H= 507:00 +CPU= 33.365 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:22:56 STEP 508 H= 508:00 +CPU= 21.833 11:23:21 STEP 509 H= 509:00 +CPU= 24.209 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:27:11 STEP 510 H= 510:00 +CPU= 38.089 11:27:36 STEP 511 H= 511:00 +CPU= 24.058 11:28:00 STEP 512 H= 512:00 +CPU= 23.374 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:45:00 STEP 513 H= 513:00 +CPU= 37.463 11:45:25 STEP 514 H= 514:00 +CPU= 24.377 11:45:49 STEP 515 H= 515:00 +CPU= 24.165 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:47:48 STEP 516 H= 516:00 +CPU= 38.237 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000516 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000516 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000516 11:48:09 STEP 517 H= 517:00 +CPU= 20.493 11:48:30 STEP 518 H= 518:00 +CPU= 21.151 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:50:15 STEP 519 H= 519:00 +CPU= 32.775 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:51:46 STEP 520 H= 520:00 +CPU= 21.010 11:52:08 STEP 521 H= 521:00 +CPU= 21.284 11:52:40 STEP 522 H= 522:00 +CPU= 32.683 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:54:33 STEP 523 H= 523:00 +CPU= 20.449 11:54:55 STEP 524 H= 524:00 +CPU= 21.525 11:55:27 STEP 525 H= 525:00 +CPU= 32.081 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:56:39 STEP 526 H= 526:00 +CPU= 21.366 11:57:00 STEP 527 H= 527:00 +CPU= 20.729 Suspend request received from the BOINC client, suspending the child process Resuming the child process 11:59:00 STEP 528 H= 528:00 +CPU= 38.355 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000528 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000528 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000504 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000504 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000504 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000516 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000516 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000516 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_21.zip Uploading the intermediate file: upload_file_21.zip 11:59:20 STEP 529 H= 529:00 +CPU= 20.084 11:59:41 STEP 530 H= 530:00 +CPU= 21.184 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:01:24 STEP 531 H= 531:00 +CPU= 32.531 12:01:45 STEP 532 H= 532:00 +CPU= 21.007 12:02:05 STEP 533 H= 533:00 +CPU= 19.892 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:03:50 STEP 534 H= 534:00 +CPU= 33.368 12:04:10 STEP 535 H= 535:00 +CPU= 20.325 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:05:43 STEP 536 H= 536:00 +CPU= 21.822 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:07:17 STEP 537 H= 537:00 +CPU= 33.070 12:07:38 STEP 538 H= 538:00 +CPU= 21.227 12:07:58 STEP 539 H= 539:00 +CPU= 20.311 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:09:36 STEP 540 H= 540:00 +CPU= 36.373 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000540 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000540 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000540 12:09:56 STEP 541 H= 541:00 +CPU= 20.229 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:11:40 STEP 542 H= 542:00 +CPU= 21.334 12:12:13 STEP 543 H= 543:00 +CPU= 33.088 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:13:35 STEP 544 H= 544:00 +CPU= 21.061 12:13:56 STEP 545 H= 545:00 +CPU= 20.965 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:15:41 STEP 546 H= 546:00 +CPU= 33.339 12:16:02 STEP 547 H= 547:00 +CPU= 21.009 12:16:23 STEP 548 H= 548:00 +CPU= 20.789 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:18:16 STEP 549 H= 549:00 +CPU= 31.885 12:18:38 STEP 550 H= 550:00 +CPU= 21.834 12:18:59 STEP 551 H= 551:00 +CPU= 20.591 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:21:19 STEP 552 H= 552:00 +CPU= 38.485 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000552 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000552 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000528 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000540 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000540 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000540 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_22.zip Uploading the intermediate file: upload_file_22.zip 12:21:39 STEP 553 H= 553:00 +CPU= 20.076 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:23:38 STEP 554 H= 554:00 +CPU= 21.134 12:24:11 STEP 555 H= 555:00 +CPU= 33.311 12:24:32 STEP 556 H= 556:00 +CPU= 20.932 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:26:23 STEP 557 H= 557:00 +CPU= 20.334 12:26:56 STEP 558 H= 558:00 +CPU= 33.429 12:27:20 STEP 559 H= 559:00 +CPU= 23.173 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:33:07 STEP 560 H= 560:00 +CPU= 24.333 12:33:45 STEP 561 H= 561:00 +CPU= 37.185 Suspend request received from the BOINC client, suspending the child process Resuming the child process 12:52:12 STEP 562 H= 562:00 +CPU= 23.314 12:52:35 STEP 563 H= 563:00 +CPU= 22.504 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:00:21 STEP 564 H= 564:00 +CPU= 42.707 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000564 13:00:45 STEP 565 H= 565:00 +CPU= 23.249 13:01:09 STEP 566 H= 566:00 +CPU= 23.722 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0096 (argv4) batchid: 958 (argv5) wuid: 12174740 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/4 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_02bb1386e77f1f6a5098ca4ee71ac874 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip ic_ancil_file: ic_ancil_12174740 ifsdata_file: ifsdata_12174740 climate_data_file: clim_data_12174740 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_3d423343c5e75ef253b31f596e3dabad to: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_add8a1baf742cd15861b692d5491858d to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12174740.xml -- Model is restarting -- Adjusting last_iter, 567, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml last_cpu_time: 16369 upload_file_number: 23 last_iter: 551 last_upload: 1987200 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001 The child process has been launched with process id: 3234650 Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3234650:3234650] [20221222:132036:1671682836:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:25:27 STEP 552 H= 552:00 +CPU= 40.488 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:34:31 STEP 553 H= 553:00 +CPU= 21.627 13:34:52 STEP 554 H= 554:00 +CPU= 21.382 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:36:58 STEP 555 H= 555:00 +CPU= 33.393 13:37:19 STEP 556 H= 556:00 +CPU= 21.113 13:37:39 STEP 557 H= 557:00 +CPU= 20.089 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:39:34 STEP 558 H= 558:00 +CPU= 33.522 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:41:07 STEP 559 H= 559:00 +CPU= 21.221 13:41:28 STEP 560 H= 560:00 +CPU= 20.923 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:43:22 STEP 561 H= 561:00 +CPU= 32.714 13:43:43 STEP 562 H= 562:00 +CPU= 20.593 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:45:17 STEP 563 H= 563:00 +CPU= 21.903 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:48:15 STEP 564 H= 564:00 +CPU= 35.747 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000564 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000564 13:48:37 STEP 565 H= 565:00 +CPU= 21.402 13:48:58 STEP 566 H= 566:00 +CPU= 20.648 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:50:53 STEP 567 H= 567:00 +CPU= 33.179 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:52:17 STEP 568 H= 568:00 +CPU= 21.728 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:53:31 STEP 569 H= 569:00 +CPU= 21.798 13:54:03 STEP 570 H= 570:00 +CPU= 32.041 13:54:24 STEP 571 H= 571:00 +CPU= 20.487 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:56:07 STEP 572 H= 572:00 +CPU= 21.543 13:56:39 STEP 573 H= 573:00 +CPU= 32.134 Suspend request received from the BOINC client, suspending the child process Resuming the child process 13:58:42 STEP 574 H= 574:00 +CPU= 21.596 13:59:04 STEP 575 H= 575:00 +CPU= 21.898 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:00:54 STEP 576 H= 576:00 +CPU= 38.636 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000576 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000576 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000552 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000564 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000564 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000564 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_23.zip Uploading the intermediate file: upload_file_23.zip 14:01:14 STEP 577 H= 577:00 +CPU= 20.298 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:03:01 STEP 578 H= 578:00 +CPU= 20.880 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:04:37 STEP 579 H= 579:00 +CPU= 33.417 14:04:58 STEP 580 H= 580:00 +CPU= 21.060 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:06:50 STEP 581 H= 581:00 +CPU= 20.575 14:07:24 STEP 582 H= 582:00 +CPU= 34.174 14:07:44 STEP 583 H= 583:00 +CPU= 19.928 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:10:17 STEP 584 H= 584:00 +CPU= 21.490 14:10:49 STEP 585 H= 585:00 +CPU= 32.167 14:11:11 STEP 586 H= 586:00 +CPU= 21.048 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:12:54 STEP 587 H= 587:00 +CPU= 21.436 14:13:30 STEP 588 H= 588:00 +CPU= 35.714 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000588 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000588 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000588 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:15:22 STEP 589 H= 589:00 +CPU= 20.856 14:15:44 STEP 590 H= 590:00 +CPU= 21.431 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:17:47 STEP 591 H= 591:00 +CPU= 32.771 14:18:08 STEP 592 H= 592:00 +CPU= 21.438 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:19:49 STEP 593 H= 593:00 +CPU= 19.974 14:20:23 STEP 594 H= 594:00 +CPU= 33.334 14:20:43 STEP 595 H= 595:00 +CPU= 20.138 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:22:37 STEP 596 H= 596:00 +CPU= 22.616 14:23:10 STEP 597 H= 597:00 +CPU= 32.189 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:25:23 STEP 598 H= 598:00 +CPU= 20.856 14:25:44 STEP 599 H= 599:00 +CPU= 21.602 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:27:44 STEP 600 H= 600:00 +CPU= 39.406 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000600 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000600 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000576 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000588 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000588 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000588 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_24.zip Uploading the intermediate file: upload_file_24.zip 14:28:05 STEP 601 H= 601:00 +CPU= 20.204 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:30:06 STEP 602 H= 602:00 +CPU= 21.618 14:30:38 STEP 603 H= 603:00 +CPU= 32.545 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:32:51 STEP 604 H= 604:00 +CPU= 21.057 14:33:12 STEP 605 H= 605:00 +CPU= 21.396 14:33:46 STEP 606 H= 606:00 +CPU= 33.233 14:34:07 STEP 607 H= 607:00 +CPU= 20.808 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:35:49 STEP 608 H= 608:00 +CPU= 21.471 14:36:23 STEP 609 H= 609:00 +CPU= 32.977 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:37:57 STEP 610 H= 610:00 +CPU= 22.360 14:38:18 STEP 611 H= 611:00 +CPU= 20.543 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:40:05 STEP 612 H= 612:00 +CPU= 36.205 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000612 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000612 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000612 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:42:18 STEP 613 H= 613:00 +CPU= 20.936 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:44:41 STEP 614 H= 614:00 +CPU= 21.524 14:45:15 STEP 615 H= 615:00 +CPU= 33.131 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:48:07 STEP 616 H= 616:00 +CPU= 21.543 14:48:29 STEP 617 H= 617:00 +CPU= 22.651 14:49:03 STEP 618 H= 618:00 +CPU= 33.531 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:50:45 STEP 619 H= 619:00 +CPU= 20.773 14:51:07 STEP 620 H= 620:00 +CPU= 21.432 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:53:02 STEP 621 H= 621:00 +CPU= 33.721 14:53:23 STEP 622 H= 622:00 +CPU= 20.853 Suspend request received from the BOINC client, suspending the child process Resuming the child process 14:55:05 STEP 623 H= 623:00 +CPU= 21.259 14:55:43 STEP 624 H= 624:00 +CPU= 37.760 Suspend request received from the BOINC client, suspending the child process Resuming the child process Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000624 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000624 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000600 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000612 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000612 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000612 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_25.zip Uploading the intermediate file: upload_file_25.zip 14:57:56 STEP 625 H= 625:00 +CPU= 20.847 14:58:18 STEP 626 H= 626:00 +CPU= 22.122 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:00:12 STEP 627 H= 627:00 +CPU= 33.127 15:00:35 STEP 628 H= 628:00 +CPU= 22.054 15:00:55 STEP 629 H= 629:00 +CPU= 20.635 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:03:41 STEP 630 H= 630:00 +CPU= 33.815 15:04:02 STEP 631 H= 631:00 +CPU= 20.100 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:05:23 STEP 632 H= 632:00 +CPU= 21.428 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:06:37 STEP 633 H= 633:00 +CPU= 32.371 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:08:20 STEP 634 H= 634:00 +CPU= 20.833 15:08:41 STEP 635 H= 635:00 +CPU= 21.015 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:10:37 STEP 636 H= 636:00 +CPU= 35.595 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000636 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000636 15:10:58 STEP 637 H= 637:00 +CPU= 21.630 15:11:20 STEP 638 H= 638:00 +CPU= 21.069 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:13:14 STEP 639 H= 639:00 +CPU= 32.415 15:13:35 STEP 640 H= 640:00 +CPU= 20.746 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:15:27 STEP 641 H= 641:00 +CPU= 20.667 15:16:00 STEP 642 H= 642:00 +CPU= 33.343 15:16:21 STEP 643 H= 643:00 +CPU= 20.086 15:16:42 STEP 644 H= 644:00 +CPU= 21.335 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:18:36 STEP 645 H= 645:00 +CPU= 32.549 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:20:18 STEP 646 H= 646:00 +CPU= 20.516 15:20:40 STEP 647 H= 647:00 +CPU= 22.221 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0096 (argv4) batchid: 958 (argv5) wuid: 12174740 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/4 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_02bb1386e77f1f6a5098ca4ee71ac874 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip ic_ancil_file: ic_ancil_12174740 ifsdata_file: ifsdata_12174740 climate_data_file: clim_data_12174740 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_3d423343c5e75ef253b31f596e3dabad to: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_add8a1baf742cd15861b692d5491858d to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12174740.xml -- Model is restarting -- Adjusting last_iter, 647, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml last_cpu_time: 18812 upload_file_number: 26 last_iter: 623 last_upload: 2246400 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001 The child process has been launched with process id: 3254922 Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3254922:3254922] [20221222:152439:1671690279:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 15:25:19 STEP 648 H= 648:00 +CPU= 37.406 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000648 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000648 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000624 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000636 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000636 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000636 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_26.zip Uploading the intermediate file: upload_file_26.zip 15:25:39 STEP 649 H= 649:00 +CPU= 20.007 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:27:15 STEP 650 H= 650:00 +CPU= 19.907 15:27:48 STEP 651 H= 651:00 +CPU= 32.820 15:28:09 STEP 652 H= 652:00 +CPU= 20.932 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:30:00 STEP 653 H= 653:00 +CPU= 20.299 15:30:33 STEP 654 H= 654:00 +CPU= 32.725 15:30:53 STEP 655 H= 655:00 +CPU= 19.956 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:32:56 STEP 656 H= 656:00 +CPU= 21.295 15:33:28 STEP 657 H= 657:00 +CPU= 31.927 15:33:49 STEP 658 H= 658:00 +CPU= 20.880 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:35:42 STEP 659 H= 659:00 +CPU= 20.494 15:36:18 STEP 660 H= 660:00 +CPU= 35.787 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000660 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000660 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000660 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:38:09 STEP 661 H= 661:00 +CPU= 20.634 15:38:31 STEP 662 H= 662:00 +CPU= 21.613 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:40:24 STEP 663 H= 663:00 +CPU= 32.185 15:40:45 STEP 664 H= 664:00 +CPU= 21.617 15:41:06 STEP 665 H= 665:00 +CPU= 20.202 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:43:30 STEP 666 H= 666:00 +CPU= 33.043 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:44:53 STEP 667 H= 667:00 +CPU= 20.598 15:45:14 STEP 668 H= 668:00 +CPU= 21.526 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:46:49 STEP 669 H= 669:00 +CPU= 33.305 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:48:30 STEP 670 H= 670:00 +CPU= 20.946 15:48:52 STEP 671 H= 671:00 +CPU= 21.472 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:50:52 STEP 672 H= 672:00 +CPU= 38.623 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000672 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000648 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000660 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000660 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_27.zip Uploading the intermediate file: upload_file_27.zip 15:51:12 STEP 673 H= 673:00 +CPU= 20.481 15:51:33 STEP 674 H= 674:00 +CPU= 20.865 Suspend request received from the BOINC client, suspending the child process Resuming the child process 15:53:28 STEP 675 H= 675:00 +CPU= 33.415 15:53:49 STEP 676 H= 676:00 +CPU= 20.899 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:01:22 STEP 677 H= 677:00 +CPU= 21.898 16:01:58 STEP 678 H= 678:00 +CPU= 35.196 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:12:52 STEP 679 H= 679:00 +CPU= 22.397 16:13:15 STEP 680 H= 680:00 +CPU= 22.073 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:17:41 STEP 681 H= 681:00 +CPU= 35.150 16:18:03 STEP 682 H= 682:00 +CPU= 21.762 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:23:47 STEP 683 H= 683:00 +CPU= 22.648 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:29:09 STEP 684 H= 684:00 +CPU= 39.599 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000684 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000684 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:36:32 STEP 685 H= 685:00 +CPU= 21.702 16:36:55 STEP 686 H= 686:00 +CPU= 22.428 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:43:01 STEP 687 H= 687:00 +CPU= 34.350 16:43:24 STEP 688 H= 688:00 +CPU= 23.118 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:47:27 STEP 689 H= 689:00 +CPU= 21.678 16:48:03 STEP 690 H= 690:00 +CPU= 35.242 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:51:04 STEP 691 H= 691:00 +CPU= 21.067 Suspend request received from the BOINC client, suspending the child process Resuming the child process 16:54:58 STEP 692 H= 692:00 +CPU= 22.612 16:55:33 STEP 693 H= 693:00 +CPU= 34.400 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:00:38 STEP 694 H= 694:00 +CPU= 22.767 17:01:02 STEP 695 H= 695:00 +CPU= 23.644 Suspend request received from the BOINC client, suspending the child process Resuming the child process Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:12:37 STEP 696 H= 696:00 +CPU= 40.584 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000696 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000696 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000672 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000684 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000684 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_28.zip Uploading the intermediate file: upload_file_28.zip 17:13:00 STEP 697 H= 697:00 +CPU= 22.340 17:13:22 STEP 698 H= 698:00 +CPU= 22.492 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:17:10 STEP 699 H= 699:00 +CPU= 35.083 17:17:33 STEP 700 H= 700:00 +CPU= 22.942 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:23:37 STEP 701 H= 701:00 +CPU= 21.802 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:29:45 STEP 702 H= 702:00 +CPU= 35.847 Suspend request received from the BOINC client, suspending the child process Resuming the child process Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu (argv1) start_date: 1989050100 (argv2) exptid: hq0f (argv3) unique_member_id: 0096 (argv4) batchid: 958 (argv5) wuid: 12174740 (argv6) fclen: 123 (argv7) app_name: oifs_43r3_ps (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/4 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_ps version: 1.05 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_02bb1386e77f1f6a5098ca4ee71ac874 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0096_1989050100_123_958_12174740.zip ic_ancil_file: ic_ancil_12174740 ifsdata_file: ifsdata_12174740 climate_data_file: clim_data_12174740 horiz_resolution: 159 vert_resolution: 60 grid_type: l_2 upload_interval: 24 utstep: 3600 nfrres: restart dump frequency (steps) 24 Copying IC ancils from: ../../projects/climateprediction.net/jf_3d423343c5e75ef253b31f596e3dabad to: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12174740.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_add8a1baf742cd15861b692d5491858d to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12174740.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12174740.zip Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12174740.xml -- Model is restarting -- Adjusting last_iter, 703, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12174740.xml last_cpu_time: 20263 upload_file_number: 29 last_iter: 695 last_upload: 2505600 model_completed: 0 total_length_of_simulation: 10627200 result_base_name: oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001 The child process has been launched with process id: 3274993 Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil)) [EC_DRHOOK:p500:1:1:3274993:3274993] [20221222:173603:1671698163:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:40:32 STEP 696 H= 696:00 +CPU= 39.198 17:40:55 STEP 697 H= 697:00 +CPU= 22.405 17:41:18 STEP 698 H= 698:00 +CPU= 23.187 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:47:36 STEP 699 H= 699:00 +CPU= 35.298 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:51:32 STEP 700 H= 700:00 +CPU= 23.295 17:51:54 STEP 701 H= 701:00 +CPU= 21.725 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:56:23 STEP 702 H= 702:00 +CPU= 36.151 Suspend request received from the BOINC client, suspending the child process Resuming the child process 17:59:07 STEP 703 H= 703:00 +CPU= 22.427 17:59:30 STEP 704 H= 704:00 +CPU= 22.809 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:03:06 STEP 705 H= 705:00 +CPU= 35.096 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:07:32 STEP 706 H= 706:00 +CPU= 23.068 18:07:55 STEP 707 H= 707:00 +CPU= 23.200 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:11:37 STEP 708 H= 708:00 +CPU= 38.589 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000708 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000708 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000708 18:12:00 STEP 709 H= 709:00 +CPU= 22.711 18:12:22 STEP 710 H= 710:00 +CPU= 22.103 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:18:29 STEP 711 H= 711:00 +CPU= 35.360 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:21:53 STEP 712 H= 712:00 +CPU= 22.709 18:22:16 STEP 713 H= 713:00 +CPU= 22.382 18:22:51 STEP 714 H= 714:00 +CPU= 34.855 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:25:55 STEP 715 H= 715:00 +CPU= 22.877 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:30:00 STEP 716 H= 716:00 +CPU= 21.966 18:30:37 STEP 717 H= 717:00 +CPU= 36.041 18:30:59 STEP 718 H= 718:00 +CPU= 22.213 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:37:14 STEP 719 H= 719:00 +CPU= 22.837 Suspend request received from the BOINC client, suspending the child process Resuming the child process 18:45:08 STEP 720 H= 720:00 +CPU= 40.384 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000720 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000720 Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000720 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000696 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMGGhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMSHhq0f+000708 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174740/ICMUAhq0f+000708 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0096_1989050100_123_958_12174740_0_r2061070001_29.zip Uploading the intermediate file: upload_file_29.zip 18:45:30 STEP 721 H= 721:00 +CPU= 21.982 18:45:53 STEP 722 H= 722:00 +CPU= 22.606 Abort request received from BOINC client, ending the child process </stderr_txt> ]]> |
No trickles! |
---|
©2024 climateprediction.net