<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
1674464258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:7894:7894] [20230123:095738:1674464258:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:59:28 STEP 1056 H=1056:00 +CPU= 68.673
10:00:00 STEP 1057 H=1057:00 +CPU= 30.727
10:00:25 STEP 1058 H=1058:00 +CPU= 24.828
10:01:10 STEP 1059 H=1059:00 +CPU= 43.776
10:01:35 STEP 1060 H=1060:00 +CPU= 25.105
10:02:00 STEP 1061 H=1061:00 +CPU= 24.487
10:02:47 STEP 1062 H=1062:00 +CPU= 46.716
10:03:11 STEP 1063 H=1063:00 +CPU= 24.450
10:03:38 STEP 1064 H=1064:00 +CPU= 26.110
10:04:21 STEP 1065 H=1065:00 +CPU= 42.558
10:04:45 STEP 1066 H=1066:00 +CPU= 24.584
10:05:15 STEP 1067 H=1067:00 +CPU= 29.729
10:05:59 STEP 1068 H=1068:00 +CPU= 43.414
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001068
10:06:23 STEP 1069 H=1069:00 +CPU= 24.055
10:06:53 STEP 1070 H=1070:00 +CPU= 30.299
10:07:32 STEP 1071 H=1071:00 +CPU= 37.960
10:08:01 STEP 1072 H=1072:00 +CPU= 29.149
10:08:27 STEP 1073 H=1073:00 +CPU= 25.818
10:09:06 STEP 1074 H=1074:00 +CPU= 38.488
10:09:36 STEP 1075 H=1075:00 +CPU= 29.307
10:10:00 STEP 1076 H=1076:00 +CPU= 24.626
10:10:43 STEP 1077 H=1077:00 +CPU= 42.288
10:11:09 STEP 1078 H=1078:00 +CPU= 25.461
10:11:34 STEP 1079 H=1079:00 +CPU= 24.907
10:12:24 STEP 1080 H=1080:00 +CPU= 50.777
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_44.zip
Uploading the intermediate file: upload_file_44.zip
10:12:49 STEP 1081 H=1081:00 +CPU= 24.732
10:13:17 STEP 1082 H=1082:00 +CPU= 27.742
10:14:05 STEP 1083 H=1083:00 +CPU= 45.191
10:14:36 STEP 1084 H=1084:00 +CPU= 28.909
10:15:10 STEP 1085 H=1085:00 +CPU= 33.222
10:15:49 STEP 1086 H=1086:00 +CPU= 38.898
10:16:16 STEP 1087 H=1087:00 +CPU= 26.368
10:16:46 STEP 1088 H=1088:00 +CPU= 29.575
10:17:24 STEP 1089 H=1089:00 +CPU= 37.921
10:17:57 STEP 1090 H=1090:00 +CPU= 32.231
10:18:27 STEP 1091 H=1091:00 +CPU= 29.341
10:19:29 STEP 1092 H=1092:00 +CPU= 56.027
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001092
10:19:58 STEP 1093 H=1093:00 +CPU= 27.453
10:20:27 STEP 1094 H=1094:00 +CPU= 28.534
10:21:14 STEP 1095 H=1095:00 +CPU= 44.750
10:21:45 STEP 1096 H=1096:00 +CPU= 29.318
10:22:25 STEP 1097 H=1097:00 +CPU= 35.469
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:24:52 STEP 1098 H=1098:00 +CPU= 56.450
10:25:29 STEP 1099 H=1099:00 +CPU= 32.245
10:26:05 STEP 1100 H=1100:00 +CPU= 34.032
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:27:12 STEP 1101 H=1101:00 +CPU= 49.986
10:27:46 STEP 1102 H=1102:00 +CPU= 31.514
10:28:25 STEP 1103 H=1103:00 +CPU= 34.740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:29:49 STEP 1104 H=1104:00 +CPU= 62.120
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_45.zip
Uploading the intermediate file: upload_file_45.zip
10:30:28 STEP 1105 H=1105:00 +CPU= 35.117
10:31:08 STEP 1106 H=1106:00 +CPU= 34.146
10:32:09 STEP 1107 H=1107:00 +CPU= 53.128
10:32:41 STEP 1108 H=1108:00 +CPU= 29.986
10:33:10 STEP 1109 H=1109:00 +CPU= 27.319
10:34:10 STEP 1110 H=1110:00 +CPU= 53.829
10:34:45 STEP 1111 H=1111:00 +CPU= 31.588
10:35:28 STEP 1112 H=1112:00 +CPU= 36.749
10:36:17 STEP 1113 H=1113:00 +CPU= 45.670
10:36:57 STEP 1114 H=1114:00 +CPU= 35.215
10:37:30 STEP 1115 H=1115:00 +CPU= 29.473
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:38:51 STEP 1116 H=1116:00 +CPU= 61.352
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:40:05 STEP 1117 H=1117:00 +CPU= 40.266
10:40:36 STEP 1118 H=1118:00 +CPU= 29.500
10:41:34 STEP 1119 H=1119:00 +CPU= 51.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:42:40 STEP 1120 H=1120:00 +CPU= 39.529
10:43:40 STEP 1121 H=1121:00 +CPU= 39.809
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:45:42 STEP 1122 H=1122:00 +CPU= 57.595
10:46:25 STEP 1123 H=1123:00 +CPU= 35.594
10:46:57 STEP 1124 H=1124:00 +CPU= 29.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:48:40 STEP 1125 H=1125:00 +CPU= 56.121
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:52:19 STEP 1126 H=1126:00 +CPU= 40.848
10:53:17 STEP 1127 H=1127:00 +CPU= 43.647
10:54:40 STEP 1128 H=1128:00 +CPU= 66.893
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_46.zip
Uploading the intermediate file: upload_file_46.zip
10:55:14 STEP 1129 H=1129:00 +CPU= 30.380
10:55:49 STEP 1130 H=1130:00 +CPU= 32.516
10:56:37 STEP 1131 H=1131:00 +CPU= 45.092
10:57:12 STEP 1132 H=1132:00 +CPU= 32.569
10:57:41 STEP 1133 H=1133:00 +CPU= 27.568
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0710
(argv4) batchid: 986
(argv5) wuid: 12203354
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ea45945fb33409046e6c5cfabc2c8f42 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0710_2017050100_123_986_12203354.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0710_2017050100_123_986_12203354.zip
ic_ancil_file: ic_ancil_12203354
ifsdata_file: ifsdata_12203354
climate_data_file: clim_data_12203354
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c91680245c88a5a7a009ad0b87001888 to: /var/lib/boinc-client/slots/6/ic_ancil_12203354.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12203354.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c03fa25d8e1b2672df027bcc6eca0526 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12203354.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12203354.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12203354.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12203354.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12203354.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12203354.xml
-- Model is restarting --
Adjusting last_iter, 1134, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12203354.xml
last_cpu_time: 38063
upload_file_number: 47
last_iter: 1127
last_upload: 4060800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915
The child process has been launched with process id: 17131
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:17131:17131] [20230123:105936:1674467976:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
11:00:35 STEP 1128 H=1128:00 +CPU= 51.551
11:01:05 STEP 1129 H=1129:00 +CPU= 29.700
11:01:36 STEP 1130 H=1130:00 +CPU= 29.991
11:02:19 STEP 1131 H=1131:00 +CPU= 41.109
11:02:53 STEP 1132 H=1132:00 +CPU= 33.431
11:03:19 STEP 1133 H=1133:00 +CPU= 25.822
11:04:03 STEP 1134 H=1134:00 +CPU= 42.387
11:04:31 STEP 1135 H=1135:00 +CPU= 27.389
11:04:58 STEP 1136 H=1136:00 +CPU= 26.254
11:05:46 STEP 1137 H=1137:00 +CPU= 46.948
11:06:14 STEP 1138 H=1138:00 +CPU= 27.245
11:06:42 STEP 1139 H=1139:00 +CPU= 26.908
11:07:33 STEP 1140 H=1140:00 +CPU= 49.366
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001140
11:07:58 STEP 1141 H=1141:00 +CPU= 25.197
11:08:31 STEP 1142 H=1142:00 +CPU= 31.509
11:09:11 STEP 1143 H=1143:00 +CPU= 39.665
11:09:42 STEP 1144 H=1144:00 +CPU= 29.988
11:10:12 STEP 1145 H=1145:00 +CPU= 28.738
11:10:53 STEP 1146 H=1146:00 +CPU= 40.280
11:11:24 STEP 1147 H=1147:00 +CPU= 30.374
11:11:50 STEP 1148 H=1148:00 +CPU= 25.748
11:12:37 STEP 1149 H=1149:00 +CPU= 45.189
11:13:08 STEP 1150 H=1150:00 +CPU= 29.917
11:13:41 STEP 1151 H=1151:00 +CPU= 31.329
11:14:47 STEP 1152 H=1152:00 +CPU= 60.912
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_47.zip
Uploading the intermediate file: upload_file_47.zip
11:15:17 STEP 1153 H=1153:00 +CPU= 28.875
11:15:58 STEP 1154 H=1154:00 +CPU= 37.722
11:16:48 STEP 1155 H=1155:00 +CPU= 47.694
11:17:28 STEP 1156 H=1156:00 +CPU= 37.850
11:18:01 STEP 1157 H=1157:00 +CPU= 31.273
11:18:50 STEP 1158 H=1158:00 +CPU= 48.096
11:19:19 STEP 1159 H=1159:00 +CPU= 27.877
11:19:49 STEP 1160 H=1160:00 +CPU= 29.578
11:20:39 STEP 1161 H=1161:00 +CPU= 47.649
11:21:06 STEP 1162 H=1162:00 +CPU= 26.102
11:21:49 STEP 1163 H=1163:00 +CPU= 36.537
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:23:28 STEP 1164 H=1164:00 +CPU= 66.830
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001164
11:24:10 STEP 1165 H=1165:00 +CPU= 33.827
11:24:40 STEP 1166 H=1166:00 +CPU= 29.687
11:25:25 STEP 1167 H=1167:00 +CPU= 44.208
11:25:53 STEP 1168 H=1168:00 +CPU= 26.698
11:26:28 STEP 1169 H=1169:00 +CPU= 33.642
11:27:28 STEP 1170 H=1170:00 +CPU= 54.533
11:28:01 STEP 1171 H=1171:00 +CPU= 33.332
11:28:27 STEP 1172 H=1172:00 +CPU= 25.038
11:29:07 STEP 1173 H=1173:00 +CPU= 40.359
11:29:37 STEP 1174 H=1174:00 +CPU= 29.333
11:30:03 STEP 1175 H=1175:00 +CPU= 25.527
11:31:01 STEP 1176 H=1176:00 +CPU= 57.376
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_48.zip
Uploading the intermediate file: upload_file_48.zip
11:31:26 STEP 1177 H=1177:00 +CPU= 24.691
11:31:52 STEP 1178 H=1178:00 +CPU= 25.727
11:32:39 STEP 1179 H=1179:00 +CPU= 46.312
11:33:04 STEP 1180 H=1180:00 +CPU= 25.240
11:33:33 STEP 1181 H=1181:00 +CPU= 28.379
11:34:16 STEP 1182 H=1182:00 +CPU= 42.029
11:34:41 STEP 1183 H=1183:00 +CPU= 24.605
11:35:14 STEP 1184 H=1184:00 +CPU= 33.187
11:35:53 STEP 1185 H=1185:00 +CPU= 38.599
11:36:22 STEP 1186 H=1186:00 +CPU= 28.288
11:36:50 STEP 1187 H=1187:00 +CPU= 27.357
11:37:33 STEP 1188 H=1188:00 +CPU= 42.470
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001188
11:38:06 STEP 1189 H=1189:00 +CPU= 32.826
11:38:31 STEP 1190 H=1190:00 +CPU= 25.197
11:39:15 STEP 1191 H=1191:00 +CPU= 42.054
11:39:45 STEP 1192 H=1192:00 +CPU= 29.485
11:40:11 STEP 1193 H=1193:00 +CPU= 25.643
11:41:01 STEP 1194 H=1194:00 +CPU= 47.921
11:41:27 STEP 1195 H=1195:00 +CPU= 25.734
11:41:55 STEP 1196 H=1196:00 +CPU= 26.957
11:42:45 STEP 1197 H=1197:00 +CPU= 47.623
11:43:12 STEP 1198 H=1198:00 +CPU= 26.611
11:43:44 STEP 1199 H=1199:00 +CPU= 30.505
11:44:40 STEP 1200 H=1200:00 +CPU= 54.365
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_49.zip
Uploading the intermediate file: upload_file_49.zip
11:45:08 STEP 1201 H=1201:00 +CPU= 26.578
11:45:43 STEP 1202 H=1202:00 +CPU= 33.668
11:46:23 STEP 1203 H=1203:00 +CPU= 39.616
11:46:54 STEP 1204 H=1204:00 +CPU= 29.514
11:47:25 STEP 1205 H=1205:00 +CPU= 29.929
11:48:07 STEP 1206 H=1206:00 +CPU= 41.269
11:48:41 STEP 1207 H=1207:00 +CPU= 32.675
11:49:09 STEP 1208 H=1208:00 +CPU= 26.764
11:49:53 STEP 1209 H=1209:00 +CPU= 42.769
11:50:25 STEP 1210 H=1210:00 +CPU= 31.129
11:50:52 STEP 1211 H=1211:00 +CPU= 25.857
11:51:58 STEP 1212 H=1212:00 +CPU= 59.818
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001212
11:52:33 STEP 1213 H=1213:00 +CPU= 31.923
11:53:15 STEP 1214 H=1214:00 +CPU= 38.591
11:54:02 STEP 1215 H=1215:00 +CPU= 44.142
11:54:41 STEP 1216 H=1216:00 +CPU= 36.061
11:55:12 STEP 1217 H=1217:00 +CPU= 29.219
11:55:54 STEP 1218 H=1218:00 +CPU= 41.026
11:56:29 STEP 1219 H=1219:00 +CPU= 33.974
11:56:58 STEP 1220 H=1220:00 +CPU= 27.956
11:57:41 STEP 1221 H=1221:00 +CPU= 41.300
11:58:14 STEP 1222 H=1222:00 +CPU= 32.293
11:58:42 STEP 1223 H=1223:00 +CPU= 26.778
11:59:40 STEP 1224 H=1224:00 +CPU= 54.937
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_50.zip
Uploading the intermediate file: upload_file_50.zip
12:00:06 STEP 1225 H=1225:00 +CPU= 26.047
12:00:38 STEP 1226 H=1226:00 +CPU= 30.306
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0710
(argv4) batchid: 986
(argv5) wuid: 12203354
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ea45945fb33409046e6c5cfabc2c8f42 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0710_2017050100_123_986_12203354.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0710_2017050100_123_986_12203354.zip
ic_ancil_file: ic_ancil_12203354
ifsdata_file: ifsdata_12203354
climate_data_file: clim_data_12203354
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c91680245c88a5a7a009ad0b87001888 to: /var/lib/boinc-client/slots/6/ic_ancil_12203354.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12203354.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c03fa25d8e1b2672df027bcc6eca0526 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12203354.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12203354.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12203354.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12203354.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12203354.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12203354.xml
-- Model is restarting --
Adjusting last_iter, 1227, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12203354.xml
last_cpu_time: 41570
upload_file_number: 51
last_iter: 1223
last_upload: 4406400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915
The child process has been launched with process id: 21818
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:21818:21818] [20230123:120206:1674471726:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
12:03:02 STEP 1224 H=1224:00 +CPU= 48.896
12:03:30 STEP 1225 H=1225:00 +CPU= 27.904
12:04:04 STEP 1226 H=1226:00 +CPU= 32.108
12:04:47 STEP 1227 H=1227:00 +CPU= 42.124
12:05:20 STEP 1228 H=1228:00 +CPU= 31.376
12:05:46 STEP 1229 H=1229:00 +CPU= 25.500
12:06:35 STEP 1230 H=1230:00 +CPU= 47.098
12:07:03 STEP 1231 H=1231:00 +CPU= 27.414
12:07:30 STEP 1232 H=1232:00 +CPU= 26.183
12:08:17 STEP 1233 H=1233:00 +CPU= 45.198
12:08:43 STEP 1234 H=1234:00 +CPU= 25.778
12:09:12 STEP 1235 H=1235:00 +CPU= 27.831
12:10:04 STEP 1236 H=1236:00 +CPU= 50.007
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001236
12:10:29 STEP 1237 H=1237:00 +CPU= 25.002
12:11:03 STEP 1238 H=1238:00 +CPU= 31.891
12:11:45 STEP 1239 H=1239:00 +CPU= 40.623
12:12:15 STEP 1240 H=1240:00 +CPU= 28.977
12:12:42 STEP 1241 H=1241:00 +CPU= 25.838
12:13:31 STEP 1242 H=1242:00 +CPU= 47.323
12:14:01 STEP 1243 H=1243:00 +CPU= 29.251
12:14:29 STEP 1244 H=1244:00 +CPU= 26.945
12:15:15 STEP 1245 H=1245:00 +CPU= 45.280
12:15:42 STEP 1246 H=1246:00 +CPU= 26.161
12:16:11 STEP 1247 H=1247:00 +CPU= 27.492
12:17:08 STEP 1248 H=1248:00 +CPU= 55.659
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_51.zip
Uploading the intermediate file: upload_file_51.zip
12:17:37 STEP 1249 H=1249:00 +CPU= 27.317
12:18:10 STEP 1250 H=1250:00 +CPU= 31.495
12:18:52 STEP 1251 H=1251:00 +CPU= 40.497
12:19:24 STEP 1252 H=1252:00 +CPU= 31.536
12:19:51 STEP 1253 H=1253:00 +CPU= 25.570
12:20:46 STEP 1254 H=1254:00 +CPU= 49.647
12:21:26 STEP 1255 H=1255:00 +CPU= 34.628
12:22:02 STEP 1256 H=1256:00 +CPU= 32.584
12:23:04 STEP 1257 H=1257:00 +CPU= 54.667
12:23:31 STEP 1258 H=1258:00 +CPU= 26.267
12:24:03 STEP 1259 H=1259:00 +CPU= 31.038
12:24:50 STEP 1260 H=1260:00 +CPU= 45.351
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001260
12:25:22 STEP 1261 H=1261:00 +CPU= 30.541
12:25:49 STEP 1262 H=1262:00 +CPU= 26.753
12:26:37 STEP 1263 H=1263:00 +CPU= 46.244
12:27:06 STEP 1264 H=1264:00 +CPU= 27.901
12:27:33 STEP 1265 H=1265:00 +CPU= 26.429
12:28:23 STEP 1266 H=1266:00 +CPU= 47.566
12:28:49 STEP 1267 H=1267:00 +CPU= 25.776
12:29:25 STEP 1268 H=1268:00 +CPU= 32.416
12:30:21 STEP 1269 H=1269:00 +CPU= 49.262
12:30:49 STEP 1270 H=1270:00 +CPU= 26.937
12:31:24 STEP 1271 H=1271:00 +CPU= 33.247
12:32:13 STEP 1272 H=1272:00 +CPU= 48.082
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_52.zip
Uploading the intermediate file: upload_file_52.zip
12:32:45 STEP 1273 H=1273:00 +CPU= 30.450
12:33:13 STEP 1274 H=1274:00 +CPU= 27.133
12:34:01 STEP 1275 H=1275:00 +CPU= 45.967
12:34:30 STEP 1276 H=1276:00 +CPU= 28.273
12:34:58 STEP 1277 H=1277:00 +CPU= 27.018
12:35:46 STEP 1278 H=1278:00 +CPU= 45.779
12:36:12 STEP 1279 H=1279:00 +CPU= 25.744
12:36:43 STEP 1280 H=1280:00 +CPU= 29.973
12:37:30 STEP 1281 H=1281:00 +CPU= 44.791
12:37:55 STEP 1282 H=1282:00 +CPU= 25.297
12:38:26 STEP 1283 H=1283:00 +CPU= 30.477
12:39:10 STEP 1284 H=1284:00 +CPU= 43.245
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001284
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:40:05 STEP 1285 H=1285:00 +CPU= 36.647
12:40:37 STEP 1286 H=1286:00 +CPU= 30.046
12:41:37 STEP 1287 H=1287:00 +CPU= 52.559
12:42:13 STEP 1288 H=1288:00 +CPU= 33.002
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:43:53 STEP 1289 H=1289:00 +CPU= 37.697
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:45:58 STEP 1290 H=1290:00 +CPU= 60.301
12:46:40 STEP 1291 H=1291:00 +CPU= 36.657
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:47:33 STEP 1292 H=1292:00 +CPU= 34.775
12:48:33 STEP 1293 H=1293:00 +CPU= 53.208
12:49:07 STEP 1294 H=1294:00 +CPU= 32.368
12:49:36 STEP 1295 H=1295:00 +CPU= 28.240
12:50:37 STEP 1296 H=1296:00 +CPU= 57.075
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_53.zip
Uploading the intermediate file: upload_file_53.zip
12:51:04 STEP 1297 H=1297:00 +CPU= 26.582
12:51:37 STEP 1298 H=1298:00 +CPU= 31.728
12:52:33 STEP 1299 H=1299:00 +CPU= 50.798
12:53:06 STEP 1300 H=1300:00 +CPU= 31.422
12:53:39 STEP 1301 H=1301:00 +CPU= 31.564
12:54:34 STEP 1302 H=1302:00 +CPU= 50.931
12:55:09 STEP 1303 H=1303:00 +CPU= 33.046
12:55:41 STEP 1304 H=1304:00 +CPU= 29.627
12:56:31 STEP 1305 H=1305:00 +CPU= 47.686
12:57:00 STEP 1306 H=1306:00 +CPU= 27.147
12:57:32 STEP 1307 H=1307:00 +CPU= 31.032
12:58:22 STEP 1308 H=1308:00 +CPU= 48.786
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001308
12:58:50 STEP 1309 H=1309:00 +CPU= 27.099
12:59:24 STEP 1310 H=1310:00 +CPU= 32.584
13:00:08 STEP 1311 H=1311:00 +CPU= 42.198
13:00:42 STEP 1312 H=1312:00 +CPU= 32.783
13:01:09 STEP 1313 H=1313:00 +CPU= 26.242
13:02:00 STEP 1314 H=1314:00 +CPU= 49.247
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0710
(argv4) batchid: 986
(argv5) wuid: 12203354
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_ea45945fb33409046e6c5cfabc2c8f42 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0710_2017050100_123_986_12203354.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0710_2017050100_123_986_12203354.zip
ic_ancil_file: ic_ancil_12203354
ifsdata_file: ifsdata_12203354
climate_data_file: clim_data_12203354
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c91680245c88a5a7a009ad0b87001888 to: /var/lib/boinc-client/slots/6/ic_ancil_12203354.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12203354.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c03fa25d8e1b2672df027bcc6eca0526 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12203354.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12203354.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12203354.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12203354.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12203354.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12203354.xml
-- Model is restarting --
Adjusting last_iter, 1315, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12203354.xml
last_cpu_time: 44844
upload_file_number: 54
last_iter: 1295
last_upload: 4665600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915
The child process has been launched with process id: 24408
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ModorMusic:1:1:24408:24408] [20230123:130313:1674475393:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
13:04:09 STEP 1296 H=1296:00 +CPU= 49.857
13:04:39 STEP 1297 H=1297:00 +CPU= 28.546
13:05:09 STEP 1298 H=1298:00 +CPU= 29.139
13:05:53 STEP 1299 H=1299:00 +CPU= 42.659
13:06:31 STEP 1300 H=1300:00 +CPU= 35.377
13:06:58 STEP 1301 H=1301:00 +CPU= 26.963
13:07:49 STEP 1302 H=1302:00 +CPU= 49.265
13:08:18 STEP 1303 H=1303:00 +CPU= 28.053
13:08:46 STEP 1304 H=1304:00 +CPU= 27.354
13:09:34 STEP 1305 H=1305:00 +CPU= 46.146
13:10:02 STEP 1306 H=1306:00 +CPU= 26.776
13:10:31 STEP 1307 H=1307:00 +CPU= 28.544
13:11:24 STEP 1308 H=1308:00 +CPU= 51.517
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001308
13:11:51 STEP 1309 H=1309:00 +CPU= 25.908
13:12:24 STEP 1310 H=1310:00 +CPU= 31.725
13:13:06 STEP 1311 H=1311:00 +CPU= 41.181
13:13:38 STEP 1312 H=1312:00 +CPU= 31.271
13:14:05 STEP 1313 H=1313:00 +CPU= 26.012
13:14:53 STEP 1314 H=1314:00 +CPU= 46.776
13:15:23 STEP 1315 H=1315:00 +CPU= 29.132
13:15:52 STEP 1316 H=1316:00 +CPU= 27.257
13:16:42 STEP 1317 H=1317:00 +CPU= 48.291
13:17:09 STEP 1318 H=1318:00 +CPU= 27.069
13:17:38 STEP 1319 H=1319:00 +CPU= 27.849
13:18:41 STEP 1320 H=1320:00 +CPU= 57.990
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203354/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0710_2017050100_123_986_12203354_2_r1265223915_54.zip
Uploading the intermediate file: upload_file_54.zip
13:19:08 STEP 1321 H=1321:00 +CPU= 26.088
13:19:40 STEP 1322 H=1322:00 +CPU= 31.326
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
