<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ing the child process
Resuming the child process
13:14:31 STEP 676 H= 676:00 +CPU= 40.851
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:15:49 STEP 677 H= 677:00 +CPU= 35.298
13:16:49 STEP 678 H= 678:00 +CPU= 56.961
13:17:27 STEP 679 H= 679:00 +CPU= 36.014
13:18:10 STEP 680 H= 680:00 +CPU= 37.410
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:19:31 STEP 681 H= 681:00 +CPU= 58.676
13:20:16 STEP 682 H= 682:00 +CPU= 40.559
13:20:57 STEP 683 H= 683:00 +CPU= 36.127
13:22:01 STEP 684 H= 684:00 +CPU= 61.944
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000684
13:22:36 STEP 685 H= 685:00 +CPU= 33.151
13:23:23 STEP 686 H= 686:00 +CPU= 40.220
13:24:28 STEP 687 H= 687:00 +CPU= 58.278
13:25:13 STEP 688 H= 688:00 +CPU= 38.469
13:25:55 STEP 689 H= 689:00 +CPU= 35.670
13:27:05 STEP 690 H= 690:00 +CPU= 60.016
13:27:50 STEP 691 H= 691:00 +CPU= 37.699
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:31:00 STEP 692 H= 692:00 +CPU= 38.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:32:39 STEP 693 H= 693:00 +CPU= 64.422
13:33:39 STEP 694 H= 694:00 +CPU= 44.066
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:34:50 STEP 695 H= 695:00 +CPU= 33.301
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0199
(argv4) batchid: 950
(argv5) wuid: 12166843
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4152633e346be72cb24b80dd89e71d86 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
ic_ancil_file: ic_ancil_12166843
ifsdata_file: ifsdata_12166843
climate_data_file: clim_data_12166843
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4874e45375c3cf68713a8a84190cf399 to: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_11a6b685e2c952846d2375cf9766fde7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12166843.xml
-- Model is restarting --
Adjusting last_iter, 696, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
last_cpu_time: 31765
upload_file_number: 28
last_iter: 695
last_upload: 2419200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895
The child process has been launched with process id: 78939
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:78939:78939] [20221222:133658:1671712618:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
13:38:31 STEP 696 H= 696:00 +CPU= 77.089
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_28.zip
Uploading the intermediate file: upload_file_28.zip
13:39:23 STEP 697 H= 697:00 +CPU= 43.758
13:40:14 STEP 698 H= 698:00 +CPU= 44.601
13:41:32 STEP 699 H= 699:00 +CPU= 67.236
13:42:29 STEP 700 H= 700:00 +CPU= 46.547
13:43:12 STEP 701 H= 701:00 +CPU= 40.419
13:44:33 STEP 702 H= 702:00 +CPU= 69.901
13:45:25 STEP 703 H= 703:00 +CPU= 44.595
13:46:10 STEP 704 H= 704:00 +CPU= 41.861
13:47:27 STEP 705 H= 705:00 +CPU= 67.710
13:48:21 STEP 706 H= 706:00 +CPU= 46.332
13:49:09 STEP 707 H= 707:00 +CPU= 42.390
13:50:33 STEP 708 H= 708:00 +CPU= 75.095
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000708
13:51:25 STEP 709 H= 709:00 +CPU= 44.474
13:52:16 STEP 710 H= 710:00 +CPU= 44.602
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:55:24 STEP 711 H= 711:00 +CPU= 68.047
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
13:57:59 STEP 712 H= 712:00 +CPU= 43.852
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:00:48 STEP 713 H= 713:00 +CPU= 43.607
14:02:39 STEP 714 H= 714:00 +CPU= 82.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:06:36 STEP 715 H= 715:00 +CPU= 49.513
14:07:49 STEP 716 H= 716:00 +CPU= 52.983
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:11:34 STEP 717 H= 717:00 +CPU= 63.630
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:16:09 STEP 718 H= 718:00 +CPU= 48.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:17:20 STEP 719 H= 719:00 +CPU= 47.118
14:18:37 STEP 720 H= 720:00 +CPU= 68.803
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_29.zip
Uploading the intermediate file: upload_file_29.zip
14:19:26 STEP 721 H= 721:00 +CPU= 40.607
14:20:02 STEP 722 H= 722:00 +CPU= 34.973
14:21:15 STEP 723 H= 723:00 +CPU= 62.213
14:21:58 STEP 724 H= 724:00 +CPU= 38.446
14:22:48 STEP 725 H= 725:00 +CPU= 41.322
14:23:52 STEP 726 H= 726:00 +CPU= 59.505
14:24:32 STEP 727 H= 727:00 +CPU= 37.602
14:25:12 STEP 728 H= 728:00 +CPU= 37.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
14:27:10 STEP 729 H= 729:00 +CPU= 59.720
14:27:56 STEP 730 H= 730:00 +CPU= 39.821
14:28:45 STEP 731 H= 731:00 +CPU= 40.399
14:30:11 STEP 732 H= 732:00 +CPU= 70.396
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000732
14:31:02 STEP 733 H= 733:00 +CPU= 39.835
14:31:42 STEP 734 H= 734:00 +CPU= 36.907
14:32:48 STEP 735 H= 735:00 +CPU= 59.432
14:33:29 STEP 736 H= 736:00 +CPU= 39.932
14:34:04 STEP 737 H= 737:00 +CPU= 34.352
14:35:07 STEP 738 H= 738:00 +CPU= 60.061
14:35:42 STEP 739 H= 739:00 +CPU= 34.995
14:36:21 STEP 740 H= 740:00 +CPU= 38.464
14:37:20 STEP 741 H= 741:00 +CPU= 57.984
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0199
(argv4) batchid: 950
(argv5) wuid: 12166843
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4152633e346be72cb24b80dd89e71d86 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
ic_ancil_file: ic_ancil_12166843
ifsdata_file: ifsdata_12166843
climate_data_file: clim_data_12166843
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4874e45375c3cf68713a8a84190cf399 to: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_11a6b685e2c952846d2375cf9766fde7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12166843.xml
-- Model is restarting --
Adjusting last_iter, 741, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
last_cpu_time: 34094
upload_file_number: 30
last_iter: 719
last_upload: 2592000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895
The child process has been launched with process id: 82786
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:82786:82786] [20221222:143833:1671716313:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
14:39:49 STEP 720 H= 720:00 +CPU= 69.209
14:40:30 STEP 721 H= 721:00 +CPU= 39.159
14:41:03 STEP 722 H= 722:00 +CPU= 32.634
14:42:03 STEP 723 H= 723:00 +CPU= 58.195
14:42:38 STEP 724 H= 724:00 +CPU= 34.179
14:43:17 STEP 725 H= 725:00 +CPU= 38.617
14:44:15 STEP 726 H= 726:00 +CPU= 55.467
14:44:57 STEP 727 H= 727:00 +CPU= 39.955
14:45:30 STEP 728 H= 728:00 +CPU= 32.983
14:46:32 STEP 729 H= 729:00 +CPU= 61.183
14:47:05 STEP 730 H= 730:00 +CPU= 33.132
14:47:45 STEP 731 H= 731:00 +CPU= 38.970
14:48:48 STEP 732 H= 732:00 +CPU= 61.342
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000732
14:49:28 STEP 733 H= 733:00 +CPU= 39.575
14:50:02 STEP 734 H= 734:00 +CPU= 33.628
14:51:04 STEP 735 H= 735:00 +CPU= 61.190
14:51:39 STEP 736 H= 736:00 +CPU= 33.908
14:52:19 STEP 737 H= 737:00 +CPU= 38.520
14:53:16 STEP 738 H= 738:00 +CPU= 56.497
14:53:56 STEP 739 H= 739:00 +CPU= 39.378
14:54:30 STEP 740 H= 740:00 +CPU= 33.825
14:55:31 STEP 741 H= 741:00 +CPU= 59.816
14:56:05 STEP 742 H= 742:00 +CPU= 33.287
14:56:42 STEP 743 H= 743:00 +CPU= 37.069
14:57:52 STEP 744 H= 744:00 +CPU= 66.969
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_30.zip
Uploading the intermediate file: upload_file_30.zip
14:58:31 STEP 745 H= 745:00 +CPU= 37.208
14:59:10 STEP 746 H= 746:00 +CPU= 38.543
15:00:12 STEP 747 H= 747:00 +CPU= 59.337
15:00:49 STEP 748 H= 748:00 +CPU= 36.128
15:01:26 STEP 749 H= 749:00 +CPU= 36.243
15:02:36 STEP 750 H= 750:00 +CPU= 60.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:03:51 STEP 751 H= 751:00 +CPU= 39.532
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:05:42 STEP 752 H= 752:00 +CPU= 40.934
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:07:23 STEP 753 H= 753:00 +CPU= 67.152
15:08:05 STEP 754 H= 754:00 +CPU= 39.560
15:08:47 STEP 755 H= 755:00 +CPU= 38.502
15:09:53 STEP 756 H= 756:00 +CPU= 63.053
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000756
15:10:29 STEP 757 H= 757:00 +CPU= 35.636
15:11:08 STEP 758 H= 758:00 +CPU= 36.764
15:12:13 STEP 759 H= 759:00 +CPU= 60.482
15:12:49 STEP 760 H= 760:00 +CPU= 35.574
15:13:26 STEP 761 H= 761:00 +CPU= 36.627
15:14:25 STEP 762 H= 762:00 +CPU= 57.600
15:15:02 STEP 763 H= 763:00 +CPU= 36.980
15:15:37 STEP 764 H= 764:00 +CPU= 34.414
15:16:37 STEP 765 H= 765:00 +CPU= 58.977
15:17:13 STEP 766 H= 766:00 +CPU= 35.637
15:17:50 STEP 767 H= 767:00 +CPU= 36.466
15:19:00 STEP 768 H= 768:00 +CPU= 67.681
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_31.zip
Uploading the intermediate file: upload_file_31.zip
15:19:38 STEP 769 H= 769:00 +CPU= 36.479
15:20:13 STEP 770 H= 770:00 +CPU= 34.424
15:21:15 STEP 771 H= 771:00 +CPU= 59.632
15:21:51 STEP 772 H= 772:00 +CPU= 35.587
15:22:29 STEP 773 H= 773:00 +CPU= 36.966
15:23:27 STEP 774 H= 774:00 +CPU= 57.612
15:24:05 STEP 775 H= 775:00 +CPU= 37.471
15:24:41 STEP 776 H= 776:00 +CPU= 35.188
15:25:41 STEP 777 H= 777:00 +CPU= 59.026
15:26:17 STEP 778 H= 778:00 +CPU= 35.075
15:26:53 STEP 779 H= 779:00 +CPU= 36.329
15:27:57 STEP 780 H= 780:00 +CPU= 62.517
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000780
15:28:35 STEP 781 H= 781:00 +CPU= 37.161
15:29:11 STEP 782 H= 782:00 +CPU= 35.874
15:30:10 STEP 783 H= 783:00 +CPU= 58.600
15:30:47 STEP 784 H= 784:00 +CPU= 35.696
15:31:23 STEP 785 H= 785:00 +CPU= 35.956
15:32:22 STEP 786 H= 786:00 +CPU= 57.856
15:33:08 STEP 787 H= 787:00 +CPU= 39.308
15:34:07 STEP 788 H= 788:00 +CPU= 46.128
15:35:33 STEP 789 H= 789:00 +CPU= 66.100
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:36:42 STEP 790 H= 790:00 +CPU= 38.902
15:37:33 STEP 791 H= 791:00 +CPU= 39.894
15:38:56 STEP 792 H= 792:00 +CPU= 71.578
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_32.zip
Uploading the intermediate file: upload_file_32.zip
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0199
(argv4) batchid: 950
(argv5) wuid: 12166843
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4152633e346be72cb24b80dd89e71d86 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
ic_ancil_file: ic_ancil_12166843
ifsdata_file: ifsdata_12166843
climate_data_file: clim_data_12166843
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4874e45375c3cf68713a8a84190cf399 to: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_11a6b685e2c952846d2375cf9766fde7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12166843.xml
-- Model is restarting --
Adjusting last_iter, 793, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
last_cpu_time: 37407
upload_file_number: 33
last_iter: 791
last_upload: 2851200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895
The child process has been launched with process id: 85184
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:85184:85184] [20221222:154020:1671720020:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
15:41:45 STEP 792 H= 792:00 +CPU= 68.850
15:42:39 STEP 793 H= 793:00 +CPU= 42.207
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:43:40 STEP 794 H= 794:00 +CPU= 41.113
15:44:40 STEP 795 H= 795:00 +CPU= 56.450
15:45:26 STEP 796 H= 796:00 +CPU= 41.753
15:46:04 STEP 797 H= 797:00 +CPU= 34.723
15:47:11 STEP 798 H= 798:00 +CPU= 60.370
15:47:48 STEP 799 H= 799:00 +CPU= 36.592
15:48:31 STEP 800 H= 800:00 +CPU= 38.486
15:49:28 STEP 801 H= 801:00 +CPU= 54.633
15:50:08 STEP 802 H= 802:00 +CPU= 38.664
15:50:45 STEP 803 H= 803:00 +CPU= 35.307
15:51:50 STEP 804 H= 804:00 +CPU= 62.192
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000804
15:52:34 STEP 805 H= 805:00 +CPU= 38.666
15:53:14 STEP 806 H= 806:00 +CPU= 36.880
15:54:21 STEP 807 H= 807:00 +CPU= 60.383
15:55:04 STEP 808 H= 808:00 +CPU= 38.446
15:55:57 STEP 809 H= 809:00 +CPU= 41.321
15:56:59 STEP 810 H= 810:00 +CPU= 59.319
15:57:36 STEP 811 H= 811:00 +CPU= 36.097
15:58:13 STEP 812 H= 812:00 +CPU= 37.195
15:59:11 STEP 813 H= 813:00 +CPU= 56.058
15:59:47 STEP 814 H= 814:00 +CPU= 36.360
16:00:21 STEP 815 H= 815:00 +CPU= 33.310
16:01:33 STEP 816 H= 816:00 +CPU= 68.664
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_33.zip
Uploading the intermediate file: upload_file_33.zip
16:02:11 STEP 817 H= 817:00 +CPU= 36.661
16:02:46 STEP 818 H= 818:00 +CPU= 34.899
16:03:42 STEP 819 H= 819:00 +CPU= 54.893
16:04:23 STEP 820 H= 820:00 +CPU= 39.218
16:04:56 STEP 821 H= 821:00 +CPU= 33.532
16:05:58 STEP 822 H= 822:00 +CPU= 60.357
16:06:34 STEP 823 H= 823:00 +CPU= 34.939
16:07:13 STEP 824 H= 824:00 +CPU= 37.992
16:08:07 STEP 825 H= 825:00 +CPU= 53.889
16:08:49 STEP 826 H= 826:00 +CPU= 40.909
16:09:21 STEP 827 H= 827:00 +CPU= 31.863
16:10:29 STEP 828 H= 828:00 +CPU= 66.525
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000828
16:11:03 STEP 829 H= 829:00 +CPU= 33.703
16:11:44 STEP 830 H= 830:00 +CPU= 40.402
16:12:42 STEP 831 H= 831:00 +CPU= 56.827
16:13:21 STEP 832 H= 832:00 +CPU= 38.395
16:13:55 STEP 833 H= 833:00 +CPU= 33.289
16:14:56 STEP 834 H= 834:00 +CPU= 60.472
16:15:30 STEP 835 H= 835:00 +CPU= 33.938
16:16:06 STEP 836 H= 836:00 +CPU= 34.952
16:17:07 STEP 837 H= 837:00 +CPU= 59.430
16:17:43 STEP 838 H= 838:00 +CPU= 35.913
16:18:22 STEP 839 H= 839:00 +CPU= 38.214
16:19:30 STEP 840 H= 840:00 +CPU= 66.418
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_34.zip
Uploading the intermediate file: upload_file_34.zip
16:20:07 STEP 841 H= 841:00 +CPU= 36.783
16:20:46 STEP 842 H= 842:00 +CPU= 37.773
16:21:45 STEP 843 H= 843:00 +CPU= 57.667
16:22:23 STEP 844 H= 844:00 +CPU= 36.666
16:22:56 STEP 845 H= 845:00 +CPU= 32.921
16:23:56 STEP 846 H= 846:00 +CPU= 58.551
16:24:31 STEP 847 H= 847:00 +CPU= 34.710
16:25:13 STEP 848 H= 848:00 +CPU= 40.513
16:26:11 STEP 849 H= 849:00 +CPU= 56.974
16:26:50 STEP 850 H= 850:00 +CPU= 37.855
16:27:25 STEP 851 H= 851:00 +CPU= 34.678
16:28:29 STEP 852 H= 852:00 +CPU= 63.527
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000852
16:29:04 STEP 853 H= 853:00 +CPU= 34.276
16:29:45 STEP 854 H= 854:00 +CPU= 39.364
16:30:38 STEP 855 H= 855:00 +CPU= 52.699
16:31:18 STEP 856 H= 856:00 +CPU= 38.659
16:31:49 STEP 857 H= 857:00 +CPU= 31.329
16:32:54 STEP 858 H= 858:00 +CPU= 60.003
16:33:30 STEP 859 H= 859:00 +CPU= 34.901
16:34:09 STEP 860 H= 860:00 +CPU= 38.643
16:35:03 STEP 861 H= 861:00 +CPU= 52.928
16:35:44 STEP 862 H= 862:00 +CPU= 40.864
16:36:18 STEP 863 H= 863:00 +CPU= 33.091
16:37:29 STEP 864 H= 864:00 +CPU= 69.225
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_35.zip
Uploading the intermediate file: upload_file_35.zip
16:38:06 STEP 865 H= 865:00 +CPU= 36.135
16:38:44 STEP 866 H= 866:00 +CPU= 37.700
16:39:39 STEP 867 H= 867:00 +CPU= 53.974
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0199
(argv4) batchid: 950
(argv5) wuid: 12166843
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_4152633e346be72cb24b80dd89e71d86 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0199_1981050100_123_950_12166843.zip
ic_ancil_file: ic_ancil_12166843
ifsdata_file: ifsdata_12166843
climate_data_file: clim_data_12166843
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4874e45375c3cf68713a8a84190cf399 to: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12166843.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_11a6b685e2c952846d2375cf9766fde7 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12166843.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12166843.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12166843.xml
-- Model is restarting --
Adjusting last_iter, 868, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12166843.xml
last_cpu_time: 40837
upload_file_number: 36
last_iter: 863
last_upload: 3110400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895
The child process has been launched with process id: 87662
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:87662:87662] [20221222:164123:1671723683:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
16:42:33 STEP 864 H= 864:00 +CPU= 63.196
16:43:14 STEP 865 H= 865:00 +CPU= 39.209
16:43:47 STEP 866 H= 866:00 +CPU= 33.131
16:44:48 STEP 867 H= 867:00 +CPU= 58.798
16:45:24 STEP 868 H= 868:00 +CPU= 34.983
16:46:02 STEP 869 H= 869:00 +CPU= 37.406
16:47:00 STEP 870 H= 870:00 +CPU= 56.181
16:47:39 STEP 871 H= 871:00 +CPU= 38.781
16:48:16 STEP 872 H= 872:00 +CPU= 36.142
16:49:16 STEP 873 H= 873:00 +CPU= 58.335
16:49:52 STEP 874 H= 874:00 +CPU= 35.124
16:50:29 STEP 875 H= 875:00 +CPU= 36.726
16:51:32 STEP 876 H= 876:00 +CPU= 61.172
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000876
16:52:12 STEP 877 H= 877:00 +CPU= 38.890
16:52:49 STEP 878 H= 878:00 +CPU= 36.406
16:53:45 STEP 879 H= 879:00 +CPU= 54.969
16:54:22 STEP 880 H= 880:00 +CPU= 36.264
16:54:59 STEP 881 H= 881:00 +CPU= 37.123
16:55:58 STEP 882 H= 882:00 +CPU= 57.239
16:56:35 STEP 883 H= 883:00 +CPU= 36.680
16:57:14 STEP 884 H= 884:00 +CPU= 37.805
16:58:11 STEP 885 H= 885:00 +CPU= 56.619
16:58:47 STEP 886 H= 886:00 +CPU= 34.720
16:59:23 STEP 887 H= 887:00 +CPU= 35.851
17:00:34 STEP 888 H= 888:00 +CPU= 67.838
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12166843/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0199_1981050100_123_950_12166843_1_r500032895_36.zip
Uploading the intermediate file: upload_file_36.zip
17:01:13 STEP 889 H= 889:00 +CPU= 36.894
17:01:49 STEP 890 H= 890:00 +CPU= 35.381
17:02:49 STEP 891 H= 891:00 +CPU= 58.820
17:03:25 STEP 892 H= 892:00 +CPU= 35.349
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
