climateprediction.net home page
Task 22305368

Task 22305368

Name oifs_43r3_ps_0341_1984050100_123_953_12169985_1
Workunit 12169985
Created 4 Feb 2023, 3:08:44 UTC
Sent 4 Feb 2023, 3:09:17 UTC
Report deadline 5 Apr 2023, 3:09:17 UTC
Received 9 Feb 2023, 17:29:22 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1534946
Run time 13 hours 40 min 45 sec
CPU time 13 hours 11 min 59 sec
Validate state Invalid
Credit 9,918.72
Device peak FLOPS 5.41 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.55 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,682.13 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002052
  20:08:29 STEP 2053 H=2053:00 +CPU= 17.430
  20:08:49 STEP 2054 H=2054:00 +CPU= 17.671
  20:09:18 STEP 2055 H=2055:00 +CPU= 26.475
  20:09:38 STEP 2056 H=2056:00 +CPU= 17.722
  20:09:56 STEP 2057 H=2057:00 +CPU= 17.192
  20:10:25 STEP 2058 H=2058:00 +CPU= 26.704
  20:10:42 STEP 2059 H=2059:00 +CPU= 16.703
  20:11:00 STEP 2060 H=2060:00 +CPU= 17.179
  20:11:27 STEP 2061 H=2061:00 +CPU= 25.647
  20:11:43 STEP 2062 H=2062:00 +CPU= 15.833
  20:11:59 STEP 2063 H=2063:00 +CPU= 15.922
  20:12:28 STEP 2064 H=2064:00 +CPU= 28.581
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_85.zip
Uploading the intermediate file: upload_file_85.zip
  20:12:44 STEP 2065 H=2065:00 +CPU= 15.690
  20:13:01 STEP 2066 H=2066:00 +CPU= 16.684
  20:13:29 STEP 2067 H=2067:00 +CPU= 26.087
  20:13:47 STEP 2068 H=2068:00 +CPU= 17.141
  20:14:06 STEP 2069 H=2069:00 +CPU= 16.929
  20:14:34 STEP 2070 H=2070:00 +CPU= 26.369
  20:14:52 STEP 2071 H=2071:00 +CPU= 16.767
  20:15:10 STEP 2072 H=2072:00 +CPU= 17.360
  20:15:38 STEP 2073 H=2073:00 +CPU= 26.063
  20:15:59 STEP 2074 H=2074:00 +CPU= 18.926
  20:16:21 STEP 2075 H=2075:00 +CPU= 20.191
  20:16:57 STEP 2076 H=2076:00 +CPU= 32.725
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002076
  20:17:15 STEP 2077 H=2077:00 +CPU= 16.645
  20:17:34 STEP 2078 H=2078:00 +CPU= 17.261
  20:18:02 STEP 2079 H=2079:00 +CPU= 26.045
  20:18:20 STEP 2080 H=2080:00 +CPU= 17.265
  20:18:38 STEP 2081 H=2081:00 +CPU= 16.884
  20:19:06 STEP 2082 H=2082:00 +CPU= 26.263
  20:19:25 STEP 2083 H=2083:00 +CPU= 17.406
  20:19:44 STEP 2084 H=2084:00 +CPU= 17.502
  20:20:12 STEP 2085 H=2085:00 +CPU= 26.003
  20:20:30 STEP 2086 H=2086:00 +CPU= 17.112
  20:20:48 STEP 2087 H=2087:00 +CPU= 17.317
  20:21:21 STEP 2088 H=2088:00 +CPU= 30.887
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_86.zip
Uploading the intermediate file: upload_file_86.zip
  20:21:39 STEP 2089 H=2089:00 +CPU= 16.854
  20:21:58 STEP 2090 H=2090:00 +CPU= 17.391
  20:22:25 STEP 2091 H=2091:00 +CPU= 25.973
  20:22:43 STEP 2092 H=2092:00 +CPU= 17.272
  20:23:01 STEP 2093 H=2093:00 +CPU= 16.747
  20:23:30 STEP 2094 H=2094:00 +CPU= 26.492
  20:23:48 STEP 2095 H=2095:00 +CPU= 17.195
  20:24:06 STEP 2096 H=2096:00 +CPU= 17.162
  20:24:35 STEP 2097 H=2097:00 +CPU= 26.802
  20:24:54 STEP 2098 H=2098:00 +CPU= 17.361
  20:25:11 STEP 2099 H=2099:00 +CPU= 16.571
  20:25:43 STEP 2100 H=2100:00 +CPU= 28.994
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002100
  20:26:00 STEP 2101 H=2101:00 +CPU= 16.723
  20:26:19 STEP 2102 H=2102:00 +CPU= 17.264
  20:26:47 STEP 2103 H=2103:00 +CPU= 25.901
  20:27:05 STEP 2104 H=2104:00 +CPU= 17.202
  20:27:23 STEP 2105 H=2105:00 +CPU= 16.902
  20:27:52 STEP 2106 H=2106:00 +CPU= 26.536
  20:28:09 STEP 2107 H=2107:00 +CPU= 16.696
  20:28:28 STEP 2108 H=2108:00 +CPU= 17.452
  20:28:56 STEP 2109 H=2109:00 +CPU= 25.962
  20:29:15 STEP 2110 H=2110:00 +CPU= 17.469
  20:29:34 STEP 2111 H=2111:00 +CPU= 17.811
  20:30:07 STEP 2112 H=2112:00 +CPU= 30.953
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_87.zip
Uploading the intermediate file: upload_file_87.zip
  20:30:25 STEP 2113 H=2113:00 +CPU= 16.937
  20:30:44 STEP 2114 H=2114:00 +CPU= 17.461
  20:31:12 STEP 2115 H=2115:00 +CPU= 25.914
  20:31:30 STEP 2116 H=2116:00 +CPU= 17.054
  20:31:48 STEP 2117 H=2117:00 +CPU= 16.908
  20:32:16 STEP 2118 H=2118:00 +CPU= 26.424
  20:32:34 STEP 2119 H=2119:00 +CPU= 16.712
  20:32:52 STEP 2120 H=2120:00 +CPU= 17.391
  20:33:20 STEP 2121 H=2121:00 +CPU= 26.134
  20:33:39 STEP 2122 H=2122:00 +CPU= 17.189
  20:33:57 STEP 2123 H=2123:00 +CPU= 16.978
  20:34:27 STEP 2124 H=2124:00 +CPU= 28.479
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002124
  20:34:47 STEP 2125 H=2125:00 +CPU= 17.920
  20:35:05 STEP 2126 H=2126:00 +CPU= 17.112
  20:35:33 STEP 2127 H=2127:00 +CPU= 26.380
  20:35:53 STEP 2128 H=2128:00 +CPU= 17.853
  20:36:11 STEP 2129 H=2129:00 +CPU= 16.722
  20:36:45 STEP 2130 H=2130:00 +CPU= 29.398
  20:37:06 STEP 2131 H=2131:00 +CPU= 17.694
  20:37:27 STEP 2132 H=2132:00 +CPU= 18.286
  20:38:00 STEP 2133 H=2133:00 +CPU= 27.539
  20:38:22 STEP 2134 H=2134:00 +CPU= 18.283
  20:38:44 STEP 2135 H=2135:00 +CPU= 18.620
  20:39:23 STEP 2136 H=2136:00 +CPU= 33.225
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_88.zip
Uploading the intermediate file: upload_file_88.zip
  20:39:45 STEP 2137 H=2137:00 +CPU= 18.181
  20:40:07 STEP 2138 H=2138:00 +CPU= 18.521
  20:40:40 STEP 2139 H=2139:00 +CPU= 28.120
  20:41:02 STEP 2140 H=2140:00 +CPU= 18.419
  20:41:24 STEP 2141 H=2141:00 +CPU= 18.053
  20:41:58 STEP 2142 H=2142:00 +CPU= 28.630
  20:42:19 STEP 2143 H=2143:00 +CPU= 17.745
  20:42:40 STEP 2144 H=2144:00 +CPU= 18.227
  20:43:13 STEP 2145 H=2145:00 +CPU= 27.806
  20:43:35 STEP 2146 H=2146:00 +CPU= 18.625
  20:43:56 STEP 2147 H=2147:00 +CPU= 17.740
  20:44:32 STEP 2148 H=2148:00 +CPU= 30.503
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002148
  20:44:53 STEP 2149 H=2149:00 +CPU= 18.326
  20:45:13 STEP 2150 H=2150:00 +CPU= 18.380
  20:45:46 STEP 2151 H=2151:00 +CPU= 30.146
  20:46:05 STEP 2152 H=2152:00 +CPU= 17.853
  20:46:23 STEP 2153 H=2153:00 +CPU= 17.086
  20:46:53 STEP 2154 H=2154:00 +CPU= 26.993
  20:47:11 STEP 2155 H=2155:00 +CPU= 16.850
  20:47:29 STEP 2156 H=2156:00 +CPU= 17.455
  20:47:57 STEP 2157 H=2157:00 +CPU= 25.757
  20:48:16 STEP 2158 H=2158:00 +CPU= 17.465
  20:48:34 STEP 2159 H=2159:00 +CPU= 17.466
  20:49:07 STEP 2160 H=2160:00 +CPU= 30.886
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_89.zip
Uploading the intermediate file: upload_file_89.zip
  20:49:25 STEP 2161 H=2161:00 +CPU= 16.911
  20:49:43 STEP 2162 H=2162:00 +CPU= 17.160
  20:50:11 STEP 2163 H=2163:00 +CPU= 26.626
  20:50:30 STEP 2164 H=2164:00 +CPU= 17.285
  20:50:48 STEP 2165 H=2165:00 +CPU= 16.743
  20:51:16 STEP 2166 H=2166:00 +CPU= 26.542
  20:51:34 STEP 2167 H=2167:00 +CPU= 16.823
  20:51:53 STEP 2168 H=2168:00 +CPU= 17.418
  20:52:20 STEP 2169 H=2169:00 +CPU= 25.932
  20:52:39 STEP 2170 H=2170:00 +CPU= 17.648
  20:52:57 STEP 2171 H=2171:00 +CPU= 17.016
  20:53:28 STEP 2172 H=2172:00 +CPU= 28.990
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002172
  20:53:46 STEP 2173 H=2173:00 +CPU= 17.259
  20:54:05 STEP 2174 H=2174:00 +CPU= 17.568
  20:54:33 STEP 2175 H=2175:00 +CPU= 26.088
  20:54:51 STEP 2176 H=2176:00 +CPU= 17.169
  20:55:10 STEP 2177 H=2177:00 +CPU= 17.372
  20:55:38 STEP 2178 H=2178:00 +CPU= 26.710
  20:55:56 STEP 2179 H=2179:00 +CPU= 17.138
  20:56:15 STEP 2180 H=2180:00 +CPU= 17.286
  20:56:42 STEP 2181 H=2181:00 +CPU= 25.900
  20:57:01 STEP 2182 H=2182:00 +CPU= 17.371
  20:57:20 STEP 2183 H=2183:00 +CPU= 17.382
  20:57:53 STEP 2184 H=2184:00 +CPU= 31.043
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_90.zip
Uploading the intermediate file: upload_file_90.zip
  20:58:11 STEP 2185 H=2185:00 +CPU= 17.251
  20:58:29 STEP 2186 H=2186:00 +CPU= 17.431
  20:58:58 STEP 2187 H=2187:00 +CPU= 26.628
  20:59:16 STEP 2188 H=2188:00 +CPU= 17.336
  20:59:34 STEP 2189 H=2189:00 +CPU= 16.852
  21:00:03 STEP 2190 H=2190:00 +CPU= 26.739
  21:00:21 STEP 2191 H=2191:00 +CPU= 16.928
  21:00:39 STEP 2192 H=2192:00 +CPU= 17.208
  21:01:07 STEP 2193 H=2193:00 +CPU= 26.262
  21:01:26 STEP 2194 H=2194:00 +CPU= 17.341
  21:01:45 STEP 2195 H=2195:00 +CPU= 17.398
  21:02:17 STEP 2196 H=2196:00 +CPU= 29.598
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002196
  21:02:33 STEP 2197 H=2197:00 +CPU= 15.951
  21:02:53 STEP 2198 H=2198:00 +CPU= 18.172
  21:03:24 STEP 2199 H=2199:00 +CPU= 27.849
  21:03:42 STEP 2200 H=2200:00 +CPU= 17.035
  21:03:58 STEP 2201 H=2201:00 +CPU= 16.055
  21:04:27 STEP 2202 H=2202:00 +CPU= 26.820
  21:04:43 STEP 2203 H=2203:00 +CPU= 16.380
  21:05:02 STEP 2204 H=2204:00 +CPU= 17.416
  21:05:37 STEP 2205 H=2205:00 +CPU= 29.854
  21:05:59 STEP 2206 H=2206:00 +CPU= 19.176
  21:06:20 STEP 2207 H=2207:00 +CPU= 18.141
  21:07:02 STEP 2208 H=2208:00 +CPU= 36.516
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_91.zip
Uploading the intermediate file: upload_file_91.zip
  21:07:24 STEP 2209 H=2209:00 +CPU= 18.931
  21:07:45 STEP 2210 H=2210:00 +CPU= 17.930
  21:08:17 STEP 2211 H=2211:00 +CPU= 28.304
  21:08:34 STEP 2212 H=2212:00 +CPU= 16.626
  21:08:59 STEP 2213 H=2213:00 +CPU= 21.401
  21:09:39 STEP 2214 H=2214:00 +CPU= 33.282
  21:10:02 STEP 2215 H=2215:00 +CPU= 19.241
  21:10:26 STEP 2216 H=2216:00 +CPU= 20.529
  21:11:02 STEP 2217 H=2217:00 +CPU= 29.790
  21:11:24 STEP 2218 H=2218:00 +CPU= 19.379
  21:11:44 STEP 2219 H=2219:00 +CPU= 17.774
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.433] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                STOCHPERT_LAYER 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211205:1675735925:4181.434] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 SPPTEN 
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211206:1675735926:4182.434] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:1842:1842] [20230206:211206:1675735926:4182.434] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DAFC79  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000010C2A4A  sppten_                   402  sppten.F90
oifs_43r3_model.e  00000000010BD01C  stochpert_layer_          159  stochpert_layer.F90
oifs_43r3_model.e  0000000000F251B4  callpar_                 1554  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0341
(argv4) batchid: 953
(argv5) wuid: 12169985
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_38279fae1f9f0ea8f8b2cdb2c22501fa to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0341_1984050100_123_953_12169985.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0341_1984050100_123_953_12169985.zip
ic_ancil_file: ic_ancil_12169985
ifsdata_file: ifsdata_12169985
climate_data_file: clim_data_12169985
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5c1a2a2a679c01c7469d85baae70bf9d to: /var/lib/boinc-client/slots/1/ic_ancil_12169985.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12169985.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b17ffc37b4705763ba178418e18f1000 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12169985.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12169985.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12169985.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12169985.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12169985.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12169985.xml
-- Model is restarting --
Adjusting last_iter, 2220, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12169985.xml
last_cpu_time: 45402
upload_file_number: 92
last_iter: 2207
last_upload: 7948800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913
The child process has been launched with process id: 1888
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:1888:1888] [20230207:180959:1675811399:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:10:40 STEP 2208 H=2208:00 +CPU= 33.070
  18:10:55 STEP 2209 H=2209:00 +CPU= 15.134
  18:11:13 STEP 2210 H=2210:00 +CPU= 17.952
  18:11:37 STEP 2211 H=2211:00 +CPU= 23.479
  18:11:52 STEP 2212 H=2212:00 +CPU= 15.543
  18:12:07 STEP 2213 H=2213:00 +CPU= 15.075
  18:12:31 STEP 2214 H=2214:00 +CPU= 23.991
  18:12:47 STEP 2215 H=2215:00 +CPU= 15.199
  18:13:03 STEP 2216 H=2216:00 +CPU= 16.237
  18:13:28 STEP 2217 H=2217:00 +CPU= 24.053
  18:13:45 STEP 2218 H=2218:00 +CPU= 16.185
  18:14:00 STEP 2219 H=2219:00 +CPU= 15.611
  18:14:28 STEP 2220 H=2220:00 +CPU= 26.658
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002220
  18:14:44 STEP 2221 H=2221:00 +CPU= 15.609
  18:15:01 STEP 2222 H=2222:00 +CPU= 16.232
  18:15:25 STEP 2223 H=2223:00 +CPU= 24.082
  18:15:42 STEP 2224 H=2224:00 +CPU= 16.087
  18:15:58 STEP 2225 H=2225:00 +CPU= 15.641
  18:16:23 STEP 2226 H=2226:00 +CPU= 24.469
  18:16:39 STEP 2227 H=2227:00 +CPU= 15.500
  18:16:55 STEP 2228 H=2228:00 +CPU= 16.002
  18:17:20 STEP 2229 H=2229:00 +CPU= 24.039
  18:17:37 STEP 2230 H=2230:00 +CPU= 16.011
  18:17:53 STEP 2231 H=2231:00 +CPU= 16.116
  18:18:27 STEP 2232 H=2232:00 +CPU= 33.025
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_92.zip
Uploading the intermediate file: upload_file_92.zip
  18:18:51 STEP 2233 H=2233:00 +CPU= 21.640
  18:19:14 STEP 2234 H=2234:00 +CPU= 21.335
  18:19:50 STEP 2235 H=2235:00 +CPU= 33.403
  18:20:13 STEP 2236 H=2236:00 +CPU= 22.259
  18:20:34 STEP 2237 H=2237:00 +CPU= 20.027
  18:21:09 STEP 2238 H=2238:00 +CPU= 33.046
  18:21:32 STEP 2239 H=2239:00 +CPU= 21.299
  18:21:54 STEP 2240 H=2240:00 +CPU= 20.892
  18:22:27 STEP 2241 H=2241:00 +CPU= 31.479
  18:22:49 STEP 2242 H=2242:00 +CPU= 20.880
  18:23:11 STEP 2243 H=2243:00 +CPU= 20.876
  18:23:49 STEP 2244 H=2244:00 +CPU= 35.260
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002244
  18:24:11 STEP 2245 H=2245:00 +CPU= 20.461
  18:24:33 STEP 2246 H=2246:00 +CPU= 21.012
  18:25:06 STEP 2247 H=2247:00 +CPU= 30.845
  18:25:28 STEP 2248 H=2248:00 +CPU= 20.862
  18:25:49 STEP 2249 H=2249:00 +CPU= 19.809
  18:26:21 STEP 2250 H=2250:00 +CPU= 31.029
  18:26:42 STEP 2251 H=2251:00 +CPU= 19.758
  18:27:04 STEP 2252 H=2252:00 +CPU= 20.254
  18:27:36 STEP 2253 H=2253:00 +CPU= 30.204
  18:27:57 STEP 2254 H=2254:00 +CPU= 20.145
  18:28:19 STEP 2255 H=2255:00 +CPU= 20.903
  18:29:01 STEP 2256 H=2256:00 +CPU= 38.451
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913_93.zip
Uploading the intermediate file: upload_file_93.zip
  18:29:22 STEP 2257 H=2257:00 +CPU= 19.678
  18:29:43 STEP 2258 H=2258:00 +CPU= 19.998
  18:30:16 STEP 2259 H=2259:00 +CPU= 30.930
  18:30:36 STEP 2260 H=2260:00 +CPU= 18.575
  18:30:55 STEP 2261 H=2261:00 +CPU= 18.335
  18:31:26 STEP 2262 H=2262:00 +CPU= 28.957
  18:31:43 STEP 2263 H=2263:00 +CPU= 16.835
  18:32:02 STEP 2264 H=2264:00 +CPU= 18.055
  18:32:31 STEP 2265 H=2265:00 +CPU= 27.717
  18:32:51 STEP 2266 H=2266:00 +CPU= 18.702
  18:33:11 STEP 2267 H=2267:00 +CPU= 18.388
  18:33:44 STEP 2268 H=2268:00 +CPU= 31.723
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002268
  18:34:05 STEP 2269 H=2269:00 +CPU= 19.778
  18:34:28 STEP 2270 H=2270:00 +CPU= 21.325
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0341
(argv4) batchid: 953
(argv5) wuid: 12169985
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_38279fae1f9f0ea8f8b2cdb2c22501fa to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0341_1984050100_123_953_12169985.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0341_1984050100_123_953_12169985.zip
ic_ancil_file: ic_ancil_12169985
ifsdata_file: ifsdata_12169985
climate_data_file: clim_data_12169985
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5c1a2a2a679c01c7469d85baae70bf9d to: /var/lib/boinc-client/slots/1/ic_ancil_12169985.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12169985.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b17ffc37b4705763ba178418e18f1000 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12169985.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12169985.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12169985.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12169985.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12169985.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12169985.xml
-- Model is restarting --
Adjusting last_iter, 2271, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12169985.xml
last_cpu_time: 46815
upload_file_number: 94
last_iter: 2255
last_upload: 8121600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913
The child process has been launched with process id: 5517
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183550:1675812950:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  18:36:28 STEP 2256 H=2256:00 +CPU= 33.942
  18:36:49 STEP 2257 H=2257:00 +CPU= 19.490
  18:37:10 STEP 2258 H=2258:00 +CPU= 19.317
  18:37:42 STEP 2259 H=2259:00 +CPU= 30.578
  18:38:01 STEP 2260 H=2260:00 +CPU= 17.869
  18:38:21 STEP 2261 H=2261:00 +CPU= 18.793
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2549MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
double free or corruption (top)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2549 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          WAMODEL 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           >OMP-WAMODEL 2            (1431) 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            IMPLSCH 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             SDISSIP 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183848:1675813128:178.821] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              SDISSIP_JAN 
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183849:1675813129:179.822] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:htpc:1:1:5517:5517] [20230207:183849:1675813129:179.822] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001A9D65F  sdissip_jan_              157  sdissip_jan.F
oifs_43r3_model.e  0000000001A85F63  sdissip_                   93  sdissip.F90
oifs_43r3_model.e  0000000001A3136A  implsch_                  291  implsch.F
oifs_43r3_model.e  000000000168CBD7  wamodel_                 1008  wamodel.F
oifs_43r3_model.e  0000000000872295  wavemdl_                  584  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0341
(argv4) batchid: 953
(argv5) wuid: 12169985
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12169985
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_38279fae1f9f0ea8f8b2cdb2c22501fa to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0341_1984050100_123_953_12169985.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0341_1984050100_123_953_12169985.zip
ic_ancil_file: ic_ancil_12169985
ifsdata_file: ifsdata_12169985
climate_data_file: clim_data_12169985
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5c1a2a2a679c01c7469d85baae70bf9d to: /var/lib/boinc-client/slots/1/ic_ancil_12169985.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12169985.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b17ffc37b4705763ba178418e18f1000 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12169985.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12169985.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12169985.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12169985.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12169985.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12169985.xml
-- Model is restarting --
Adjusting last_iter, 2262, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12169985.xml
last_cpu_time: 46974
upload_file_number: 94
last_iter: 2255
last_upload: 8121600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0341_1984050100_123_953_12169985_1_r2092813913
The child process has been launched with process id: 2131
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:htpc:1:1:2131:2131] [20230209:121611:1675962971:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:16:51 STEP 2256 H=2256:00 +CPU= 33.948
  12:17:06 STEP 2257 H=2257:00 +CPU= 15.073
  12:17:25 STEP 2258 H=2258:00 +CPU= 18.167
  12:17:49 STEP 2259 H=2259:00 +CPU= 23.638
  12:18:04 STEP 2260 H=2260:00 +CPU= 15.507
  12:18:19 STEP 2261 H=2261:00 +CPU= 15.128
  12:18:44 STEP 2262 H=2262:00 +CPU= 24.451
  12:19:02 STEP 2263 H=2263:00 +CPU= 17.349
  12:19:20 STEP 2264 H=2264:00 +CPU= 16.940
  12:19:47 STEP 2265 H=2265:00 +CPU= 26.009
  12:20:06 STEP 2266 H=2266:00 +CPU= 17.340
  12:20:24 STEP 2267 H=2267:00 +CPU= 17.501
  12:20:53 STEP 2268 H=2268:00 +CPU= 27.784
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002268
  12:21:12 STEP 2269 H=2269:00 +CPU= 17.416
  12:21:38 STEP 2270 H=2270:00 +CPU= 21.559
  12:22:08 STEP 2271 H=2271:00 +CPU= 27.549
  12:22:29 STEP 2272 H=2272:00 +CPU= 19.675
  12:22:53 STEP 2273 H=2273:00 +CPU= 20.878
  12:23:36 STEP 2274 H=2274:00 +CPU= 33.428
  12:24:02 STEP 2275 H=2275:00 +CPU= 21.744
  12:24:26 STEP 2276 H=2276:00 +CPU= 21.554
  12:25:03 STEP 2277 H=2277:00 +CPU= 32.670
  12:25:26 STEP 2278 H=2278:00 +CPU= 21.120
  12:25:48 STEP 2279 H=2279:00 +CPU= 21.201
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:48:26 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 7,084,800 39,834 0.0056
09 May 2023 16:48:24 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 5,875,200 33,089 0.0056
09 May 2023 16:48:23 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 5,011,200 28,678 0.0057
09 May 2023 16:48:22 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 4,147,200 24,265 0.0059
09 May 2023 16:48:19 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 3,283,200 19,855 0.0060
09 May 2023 16:48:18 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 2,419,200 15,443 0.0064
09 May 2023 16:48:16 1534946 22305368 oifs_43r3_ps_0341_1984050100_123_953_12169985_1 1,468,800 9,789 0.0067


©2024 cpdn.org