climateprediction.net home page
Task 22286718

Task 22286718

Name oifs_43r3_ps_0845_2014050100_123_983_12200489_0
Workunit 12200489
Created 24 Dec 2022, 11:13:55 UTC
Sent 5 Jan 2023, 21:00:23 UTC
Report deadline 4 Feb 2023, 21:00:23 UTC
Received 6 Jan 2023, 23:42:38 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1459164
Run time 21 hours 54 min 12 sec
CPU time 21 hours 41 min 14 sec
Validate state Invalid
Credit 12,216.96
Device peak FLOPS 4.76 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,793.63 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,598.73 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ectory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_88.zip
Uploading the intermediate file: upload_file_88.zip
  13:45:45 STEP 2137 H=2137:00 +CPU= 21.818
  13:46:09 STEP 2138 H=2138:00 +CPU= 23.100
  13:46:47 STEP 2139 H=2139:00 +CPU= 38.038
  13:47:09 STEP 2140 H=2140:00 +CPU= 22.769
  13:47:36 STEP 2141 H=2141:00 +CPU= 26.168
  13:48:15 STEP 2142 H=2142:00 +CPU= 39.859
  13:48:38 STEP 2143 H=2143:00 +CPU= 22.499
  13:48:59 STEP 2144 H=2144:00 +CPU= 20.797
  13:49:32 STEP 2145 H=2145:00 +CPU= 33.130
  13:50:00 STEP 2146 H=2146:00 +CPU= 28.567
  13:50:29 STEP 2147 H=2147:00 +CPU= 28.095
  13:51:09 STEP 2148 H=2148:00 +CPU= 40.283
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002148
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002148
  13:51:31 STEP 2149 H=2149:00 +CPU= 21.899
  13:51:55 STEP 2150 H=2150:00 +CPU= 23.967
  13:52:29 STEP 2151 H=2151:00 +CPU= 34.315
  13:52:50 STEP 2152 H=2152:00 +CPU= 20.675
  13:53:17 STEP 2153 H=2153:00 +CPU= 27.483
  13:53:51 STEP 2154 H=2154:00 +CPU= 33.659
  13:54:11 STEP 2155 H=2155:00 +CPU= 19.906
  13:54:36 STEP 2156 H=2156:00 +CPU= 24.660
  13:55:17 STEP 2157 H=2157:00 +CPU= 41.479
  13:55:39 STEP 2158 H=2158:00 +CPU= 22.342
  13:56:02 STEP 2159 H=2159:00 +CPU= 22.299
  13:56:42 STEP 2160 H=2160:00 +CPU= 39.836
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002160
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002148
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002148
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_89.zip
Uploading the intermediate file: upload_file_89.zip
  13:57:01 STEP 2161 H=2161:00 +CPU= 19.632
  13:57:25 STEP 2162 H=2162:00 +CPU= 23.447
  13:57:56 STEP 2163 H=2163:00 +CPU= 31.212
  13:58:17 STEP 2164 H=2164:00 +CPU= 20.903
  13:58:36 STEP 2165 H=2165:00 +CPU= 19.183
  13:59:07 STEP 2166 H=2166:00 +CPU= 30.998
  13:59:31 STEP 2167 H=2167:00 +CPU= 23.987
  13:59:59 STEP 2168 H=2168:00 +CPU= 28.114
  14:00:41 STEP 2169 H=2169:00 +CPU= 41.276
  14:01:07 STEP 2170 H=2170:00 +CPU= 26.643
  14:01:32 STEP 2171 H=2171:00 +CPU= 24.818
  14:02:16 STEP 2172 H=2172:00 +CPU= 43.571
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002172
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002172
  14:02:44 STEP 2173 H=2173:00 +CPU= 28.781
  14:03:08 STEP 2174 H=2174:00 +CPU= 23.052
  14:03:45 STEP 2175 H=2175:00 +CPU= 37.331
  14:04:06 STEP 2176 H=2176:00 +CPU= 21.085
  14:04:27 STEP 2177 H=2177:00 +CPU= 20.678
  14:05:03 STEP 2178 H=2178:00 +CPU= 36.662
  14:05:23 STEP 2179 H=2179:00 +CPU= 19.389
  14:05:46 STEP 2180 H=2180:00 +CPU= 23.081
  14:06:19 STEP 2181 H=2181:00 +CPU= 33.117
  14:06:40 STEP 2182 H=2182:00 +CPU= 20.610
  14:06:59 STEP 2183 H=2183:00 +CPU= 19.730
  14:07:37 STEP 2184 H=2184:00 +CPU= 37.843
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002184
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002160
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002172
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002172
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_90.zip
Uploading the intermediate file: upload_file_90.zip
Uploading trickle at timestep: 7862400
  14:08:05 STEP 2185 H=2185:00 +CPU= 27.819
  14:08:26 STEP 2186 H=2186:00 +CPU= 21.102
  14:08:59 STEP 2187 H=2187:00 +CPU= 33.121
  14:09:22 STEP 2188 H=2188:00 +CPU= 22.368
  14:09:41 STEP 2189 H=2189:00 +CPU= 19.133
  14:10:13 STEP 2190 H=2190:00 +CPU= 32.593
  14:10:36 STEP 2191 H=2191:00 +CPU= 22.847
  14:10:58 STEP 2192 H=2192:00 +CPU= 21.690
  14:11:35 STEP 2193 H=2193:00 +CPU= 36.853
  14:12:00 STEP 2194 H=2194:00 +CPU= 24.790
  14:12:24 STEP 2195 H=2195:00 +CPU= 24.647
  14:13:01 STEP 2196 H=2196:00 +CPU= 36.401
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002196
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002196
  14:13:24 STEP 2197 H=2197:00 +CPU= 23.151
  14:13:48 STEP 2198 H=2198:00 +CPU= 23.934
  14:14:20 STEP 2199 H=2199:00 +CPU= 32.034
  14:14:40 STEP 2200 H=2200:00 +CPU= 20.178
  14:15:01 STEP 2201 H=2201:00 +CPU= 20.880
  14:15:34 STEP 2202 H=2202:00 +CPU= 33.548
  14:15:54 STEP 2203 H=2203:00 +CPU= 19.654
  14:16:14 STEP 2204 H=2204:00 +CPU= 20.177
  14:16:45 STEP 2205 H=2205:00 +CPU= 30.750
  14:17:08 STEP 2206 H=2206:00 +CPU= 22.641
  14:17:28 STEP 2207 H=2207:00 +CPU= 20.662
  14:18:13 STEP 2208 H=2208:00 +CPU= 44.649
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002208
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002184
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002196
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002196
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_91.zip
Uploading the intermediate file: upload_file_91.zip
  14:18:33 STEP 2209 H=2209:00 +CPU= 19.748
  14:18:54 STEP 2210 H=2210:00 +CPU= 21.716
  14:19:37 STEP 2211 H=2211:00 +CPU= 42.036
  14:20:05 STEP 2212 H=2212:00 +CPU= 28.135
  14:20:28 STEP 2213 H=2213:00 +CPU= 23.744
  14:21:02 STEP 2214 H=2214:00 +CPU= 33.575
  14:21:22 STEP 2215 H=2215:00 +CPU= 19.551
  14:21:43 STEP 2216 H=2216:00 +CPU= 21.023
  14:22:21 STEP 2217 H=2217:00 +CPU= 38.623
  14:22:44 STEP 2218 H=2218:00 +CPU= 22.715
  14:23:08 STEP 2219 H=2219:00 +CPU= 23.746
  14:23:53 STEP 2220 H=2220:00 +CPU= 44.837
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002220
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002220
  14:24:15 STEP 2221 H=2221:00 +CPU= 22.197
  14:24:39 STEP 2222 H=2222:00 +CPU= 23.699
  14:25:14 STEP 2223 H=2223:00 +CPU= 35.725
  14:25:39 STEP 2224 H=2224:00 +CPU= 24.754
  14:25:59 STEP 2225 H=2225:00 +CPU= 20.085
  14:26:34 STEP 2226 H=2226:00 +CPU= 35.252
  14:26:54 STEP 2227 H=2227:00 +CPU= 19.372
  14:27:14 STEP 2228 H=2228:00 +CPU= 20.652
  14:27:50 STEP 2229 H=2229:00 +CPU= 35.657
  14:28:10 STEP 2230 H=2230:00 +CPU= 20.209
  14:28:31 STEP 2231 H=2231:00 +CPU= 20.639
  14:29:07 STEP 2232 H=2232:00 +CPU= 35.651
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002232
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002208
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002220
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002220
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_92.zip
Uploading the intermediate file: upload_file_92.zip
  14:29:31 STEP 2233 H=2233:00 +CPU= 24.081
  14:29:55 STEP 2234 H=2234:00 +CPU= 24.121
  14:30:30 STEP 2235 H=2235:00 +CPU= 35.092
  14:30:51 STEP 2236 H=2236:00 +CPU= 21.208
  14:31:11 STEP 2237 H=2237:00 +CPU= 19.825
  14:31:50 STEP 2238 H=2238:00 +CPU= 39.429
  14:32:11 STEP 2239 H=2239:00 +CPU= 20.872
  14:32:34 STEP 2240 H=2240:00 +CPU= 22.804
  14:33:05 STEP 2241 H=2241:00 +CPU= 31.278
  14:33:27 STEP 2242 H=2242:00 +CPU= 21.611
  14:33:47 STEP 2243 H=2243:00 +CPU= 19.840
  14:34:27 STEP 2244 H=2244:00 +CPU= 39.756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002244
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002244
  14:34:46 STEP 2245 H=2245:00 +CPU= 19.606
  14:35:12 STEP 2246 H=2246:00 +CPU= 25.571
  14:35:48 STEP 2247 H=2247:00 +CPU= 35.903
  14:36:12 STEP 2248 H=2248:00 +CPU= 23.884
  14:36:35 STEP 2249 H=2249:00 +CPU= 23.648
  14:37:13 STEP 2250 H=2250:00 +CPU= 37.371
  14:37:38 STEP 2251 H=2251:00 +CPU= 25.548
  14:38:03 STEP 2252 H=2252:00 +CPU= 24.593
  14:38:38 STEP 2253 H=2253:00 +CPU= 35.396
  14:39:00 STEP 2254 H=2254:00 +CPU= 21.745
  14:39:22 STEP 2255 H=2255:00 +CPU= 21.654
  14:39:58 STEP 2256 H=2256:00 +CPU= 36.150
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_93.zip
Uploading the intermediate file: upload_file_93.zip
  14:40:19 STEP 2257 H=2257:00 +CPU= 20.668
  14:40:41 STEP 2258 H=2258:00 +CPU= 22.303
  14:41:13 STEP 2259 H=2259:00 +CPU= 32.509
  14:41:34 STEP 2260 H=2260:00 +CPU= 20.308
  14:41:53 STEP 2261 H=2261:00 +CPU= 19.539
  14:42:26 STEP 2262 H=2262:00 +CPU= 32.768
  14:42:52 STEP 2263 H=2263:00 +CPU= 25.735
  14:43:20 STEP 2264 H=2264:00 +CPU= 28.172
  14:43:55 STEP 2265 H=2265:00 +CPU= 35.270
  14:44:24 STEP 2266 H=2266:00 +CPU= 29.048
  14:44:47 STEP 2267 H=2267:00 +CPU= 22.813
  14:45:24 STEP 2268 H=2268:00 +CPU= 37.051
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002268
  14:45:44 STEP 2269 H=2269:00 +CPU= 19.595
  14:46:04 STEP 2270 H=2270:00 +CPU= 19.940
  14:46:35 STEP 2271 H=2271:00 +CPU= 31.495
  14:46:55 STEP 2272 H=2272:00 +CPU= 20.227
  14:47:15 STEP 2273 H=2273:00 +CPU= 19.446
  14:47:47 STEP 2274 H=2274:00 +CPU= 32.234
  14:48:06 STEP 2275 H=2275:00 +CPU= 18.337
  14:48:25 STEP 2276 H=2276:00 +CPU= 19.028
  14:48:54 STEP 2277 H=2277:00 +CPU= 29.446
  14:49:13 STEP 2278 H=2278:00 +CPU= 18.959
  14:49:32 STEP 2279 H=2279:00 +CPU= 19.029
  14:50:06 STEP 2280 H=2280:00 +CPU= 34.222
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_94.zip
Uploading the intermediate file: upload_file_94.zip
  14:50:25 STEP 2281 H=2281:00 +CPU= 18.702
  14:50:44 STEP 2282 H=2282:00 +CPU= 19.009
  14:51:13 STEP 2283 H=2283:00 +CPU= 29.294
  14:51:34 STEP 2284 H=2284:00 +CPU= 20.606
  14:51:53 STEP 2285 H=2285:00 +CPU= 19.135
  14:52:25 STEP 2286 H=2286:00 +CPU= 32.351
  14:52:48 STEP 2287 H=2287:00 +CPU= 22.564
  14:53:09 STEP 2288 H=2288:00 +CPU= 20.923
  14:53:40 STEP 2289 H=2289:00 +CPU= 30.736
  14:54:00 STEP 2290 H=2290:00 +CPU= 20.431
  14:54:22 STEP 2291 H=2291:00 +CPU= 21.563
  14:55:05 STEP 2292 H=2292:00 +CPU= 43.429
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002292
  14:55:26 STEP 2293 H=2293:00 +CPU= 20.953
  14:55:50 STEP 2294 H=2294:00 +CPU= 23.607
  14:56:23 STEP 2295 H=2295:00 +CPU= 33.726
  14:56:44 STEP 2296 H=2296:00 +CPU= 20.386
  14:57:03 STEP 2297 H=2297:00 +CPU= 18.786
  14:57:36 STEP 2298 H=2298:00 +CPU= 33.095
  14:57:54 STEP 2299 H=2299:00 +CPU= 18.792
  14:58:14 STEP 2300 H=2300:00 +CPU= 19.852
  14:58:50 STEP 2301 H=2301:00 +CPU= 35.843
  14:59:12 STEP 2302 H=2302:00 +CPU= 22.319
  14:59:37 STEP 2303 H=2303:00 +CPU= 24.117
  15:00:18 STEP 2304 H=2304:00 +CPU= 41.282
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_95.zip
Uploading the intermediate file: upload_file_95.zip
  15:00:41 STEP 2305 H=2305:00 +CPU= 23.351
  15:01:02 STEP 2306 H=2306:00 +CPU= 20.419
  15:01:34 STEP 2307 H=2307:00 +CPU= 32.314
  15:01:53 STEP 2308 H=2308:00 +CPU= 19.240
  15:02:14 STEP 2309 H=2309:00 +CPU= 21.221
  15:02:48 STEP 2310 H=2310:00 +CPU= 33.127
  15:03:10 STEP 2311 H=2311:00 +CPU= 22.278
  15:03:35 STEP 2312 H=2312:00 +CPU= 25.080
  15:04:12 STEP 2313 H=2313:00 +CPU= 36.594
  15:04:34 STEP 2314 H=2314:00 +CPU= 22.622
  15:05:00 STEP 2315 H=2315:00 +CPU= 25.923
  15:05:46 STEP 2316 H=2316:00 +CPU= 45.696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002316
  15:06:08 STEP 2317 H=2317:00 +CPU= 21.644
  15:06:30 STEP 2318 H=2318:00 +CPU= 22.328
  15:07:07 STEP 2319 H=2319:00 +CPU= 37.249
  15:07:31 STEP 2320 H=2320:00 +CPU= 23.279
  15:07:51 STEP 2321 H=2321:00 +CPU= 20.140
  15:08:26 STEP 2322 H=2322:00 +CPU= 35.211
  15:08:45 STEP 2323 H=2323:00 +CPU= 19.047
  15:09:05 STEP 2324 H=2324:00 +CPU= 19.613
  15:09:37 STEP 2325 H=2325:00 +CPU= 32.110
  15:09:56 STEP 2326 H=2326:00 +CPU= 19.647
  15:10:15 STEP 2327 H=2327:00 +CPU= 19.134
  15:10:52 STEP 2328 H=2328:00 +CPU= 36.693
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_96.zip
Uploading the intermediate file: upload_file_96.zip
  15:11:12 STEP 2329 H=2329:00 +CPU= 20.091
  15:11:33 STEP 2330 H=2330:00 +CPU= 20.982
  15:12:12 STEP 2331 H=2331:00 +CPU= 38.528
  15:12:38 STEP 2332 H=2332:00 +CPU= 26.448
  15:13:00 STEP 2333 H=2333:00 +CPU= 22.053
  15:13:37 STEP 2334 H=2334:00 +CPU= 36.203
  15:14:00 STEP 2335 H=2335:00 +CPU= 23.882
  15:14:21 STEP 2336 H=2336:00 +CPU= 20.975
  15:14:54 STEP 2337 H=2337:00 +CPU= 32.661
  15:15:16 STEP 2338 H=2338:00 +CPU= 22.236
  15:15:39 STEP 2339 H=2339:00 +CPU= 22.132
  15:16:18 STEP 2340 H=2340:00 +CPU= 39.895
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002340
  15:16:38 STEP 2341 H=2341:00 +CPU= 19.286
  15:16:59 STEP 2342 H=2342:00 +CPU= 21.133
  15:17:36 STEP 2343 H=2343:00 +CPU= 37.407
  15:17:59 STEP 2344 H=2344:00 +CPU= 22.710
  15:18:20 STEP 2345 H=2345:00 +CPU= 21.162
  15:18:58 STEP 2346 H=2346:00 +CPU= 38.023
  15:19:20 STEP 2347 H=2347:00 +CPU= 22.014
  15:19:40 STEP 2348 H=2348:00 +CPU= 19.232
  15:20:15 STEP 2349 H=2349:00 +CPU= 35.402
  15:20:38 STEP 2350 H=2350:00 +CPU= 23.078
  15:21:02 STEP 2351 H=2351:00 +CPU= 23.570
  15:21:44 STEP 2352 H=2352:00 +CPU= 42.121
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_97.zip
Uploading the intermediate file: upload_file_97.zip
  15:22:03 STEP 2353 H=2353:00 +CPU= 19.233
  15:22:23 STEP 2354 H=2354:00 +CPU= 20.291
  15:22:54 STEP 2355 H=2355:00 +CPU= 30.271
  15:23:14 STEP 2356 H=2356:00 +CPU= 20.069
  15:23:33 STEP 2357 H=2357:00 +CPU= 19.488
  15:24:11 STEP 2358 H=2358:00 +CPU= 37.331
  15:24:31 STEP 2359 H=2359:00 +CPU= 20.273
  15:24:56 STEP 2360 H=2360:00 +CPU= 24.927
  15:25:31 STEP 2361 H=2361:00 +CPU= 34.745
  15:25:52 STEP 2362 H=2362:00 +CPU= 21.224
  15:26:15 STEP 2363 H=2363:00 +CPU= 22.990
  15:26:51 STEP 2364 H=2364:00 +CPU= 36.636
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002364
  15:27:11 STEP 2365 H=2365:00 +CPU= 20.055
  15:27:31 STEP 2366 H=2366:00 +CPU= 19.981
  15:28:06 STEP 2367 H=2367:00 +CPU= 34.628
  15:28:27 STEP 2368 H=2368:00 +CPU= 20.617
  15:28:48 STEP 2369 H=2369:00 +CPU= 21.736
  15:29:22 STEP 2370 H=2370:00 +CPU= 33.709
  15:29:43 STEP 2371 H=2371:00 +CPU= 20.387
  15:30:04 STEP 2372 H=2372:00 +CPU= 21.069
  15:30:36 STEP 2373 H=2373:00 +CPU= 32.162
  15:30:57 STEP 2374 H=2374:00 +CPU= 21.114
  15:31:17 STEP 2375 H=2375:00 +CPU= 20.245
  15:31:54 STEP 2376 H=2376:00 +CPU= 36.350
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_98.zip
Uploading the intermediate file: upload_file_98.zip
  15:32:13 STEP 2377 H=2377:00 +CPU= 19.348
  15:32:33 STEP 2378 H=2378:00 +CPU= 20.091
  15:33:03 STEP 2379 H=2379:00 +CPU= 29.997
  15:33:23 STEP 2380 H=2380:00 +CPU= 19.988
  15:33:48 STEP 2381 H=2381:00 +CPU= 24.983
  15:34:23 STEP 2382 H=2382:00 +CPU= 34.989
  15:34:43 STEP 2383 H=2383:00 +CPU= 20.261
  15:35:07 STEP 2384 H=2384:00 +CPU= 23.242
  15:35:42 STEP 2385 H=2385:00 +CPU= 35.649
  15:36:04 STEP 2386 H=2386:00 +CPU= 22.143
  15:36:24 STEP 2387 H=2387:00 +CPU= 19.865
  15:37:01 STEP 2388 H=2388:00 +CPU= 36.536
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002388
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002388
  15:37:21 STEP 2389 H=2389:00 +CPU= 20.130
  15:37:41 STEP 2390 H=2390:00 +CPU= 20.206
  15:38:13 STEP 2391 H=2391:00 +CPU= 32.074
  15:38:35 STEP 2392 H=2392:00 +CPU= 21.918
  15:38:55 STEP 2393 H=2393:00 +CPU= 19.740
  15:39:30 STEP 2394 H=2394:00 +CPU= 35.462
  15:39:57 STEP 2395 H=2395:00 +CPU= 26.704
  15:40:23 STEP 2396 H=2396:00 +CPU= 26.332
  15:41:01 STEP 2397 H=2397:00 +CPU= 37.906
  15:41:24 STEP 2398 H=2398:00 +CPU= 22.519
  15:41:45 STEP 2399 H=2399:00 +CPU= 20.866
  15:42:28 STEP 2400 H=2400:00 +CPU= 43.391
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_99.zip
Uploading the intermediate file: upload_file_99.zip
  15:42:55 STEP 2401 H=2401:00 +CPU= 26.800
  15:43:19 STEP 2402 H=2402:00 +CPU= 23.902
  15:44:01 STEP 2403 H=2403:00 +CPU= 42.143
  15:44:26 STEP 2404 H=2404:00 +CPU= 24.798
  15:44:49 STEP 2405 H=2405:00 +CPU= 22.794
  15:45:28 STEP 2406 H=2406:00 +CPU= 39.201
  15:45:54 STEP 2407 H=2407:00 +CPU= 26.542
  15:46:21 STEP 2408 H=2408:00 +CPU= 27.013
  15:47:02 STEP 2409 H=2409:00 +CPU= 40.211
  15:47:21 STEP 2410 H=2410:00 +CPU= 19.726
  15:47:48 STEP 2411 H=2411:00 +CPU= 26.742
  15:48:28 STEP 2412 H=2412:00 +CPU= 40.010
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002412
  15:48:54 STEP 2413 H=2413:00 +CPU= 26.142
  15:49:20 STEP 2414 H=2414:00 +CPU= 25.264
  15:49:55 STEP 2415 H=2415:00 +CPU= 35.566
  15:50:18 STEP 2416 H=2416:00 +CPU= 22.423
  15:50:43 STEP 2417 H=2417:00 +CPU= 25.815
  15:51:23 STEP 2418 H=2418:00 +CPU= 39.908
  15:51:47 STEP 2419 H=2419:00 +CPU= 23.492
  15:52:09 STEP 2420 H=2420:00 +CPU= 22.088
  15:52:43 STEP 2421 H=2421:00 +CPU= 34.211
  15:53:05 STEP 2422 H=2422:00 +CPU= 22.023
  15:53:26 STEP 2423 H=2423:00 +CPU= 20.525
  15:54:10 STEP 2424 H=2424:00 +CPU= 44.223
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_100.zip
Uploading the intermediate file: upload_file_100.zip
  15:54:30 STEP 2425 H=2425:00 +CPU= 19.925
Uploading trickle at timestep: 8726400
  15:54:50 STEP 2426 H=2426:00 +CPU= 20.530
  15:55:24 STEP 2427 H=2427:00 +CPU= 33.901
  15:55:45 STEP 2428 H=2428:00 +CPU= 21.054
  15:56:07 STEP 2429 H=2429:00 +CPU= 21.282
  15:56:43 STEP 2430 H=2430:00 +CPU= 36.152
  15:57:03 STEP 2431 H=2431:00 +CPU= 20.621
  15:57:26 STEP 2432 H=2432:00 +CPU= 22.887
  15:58:02 STEP 2433 H=2433:00 +CPU= 35.319
  15:58:25 STEP 2434 H=2434:00 +CPU= 23.541
  15:58:52 STEP 2435 H=2435:00 +CPU= 26.647
  15:59:31 STEP 2436 H=2436:00 +CPU= 39.221
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002436
  15:59:53 STEP 2437 H=2437:00 +CPU= 21.941
  16:00:15 STEP 2438 H=2438:00 +CPU= 22.054
  16:00:46 STEP 2439 H=2439:00 +CPU= 31.344
  16:01:06 STEP 2440 H=2440:00 +CPU= 20.083
  16:01:27 STEP 2441 H=2441:00 +CPU= 20.216
  16:02:01 STEP 2442 H=2442:00 +CPU= 34.623
  16:02:24 STEP 2443 H=2443:00 +CPU= 22.515
  16:02:45 STEP 2444 H=2444:00 +CPU= 21.072
  16:03:17 STEP 2445 H=2445:00 +CPU= 32.483
  16:03:39 STEP 2446 H=2446:00 +CPU= 21.729
  16:04:01 STEP 2447 H=2447:00 +CPU= 21.875
  16:04:48 STEP 2448 H=2448:00 +CPU= 46.502
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_101.zip
Uploading the intermediate file: upload_file_101.zip
  16:05:15 STEP 2449 H=2449:00 +CPU= 27.543
  16:05:39 STEP 2450 H=2450:00 +CPU= 23.881
  16:06:16 STEP 2451 H=2451:00 +CPU= 36.583
  16:06:38 STEP 2452 H=2452:00 +CPU= 21.586
  16:06:58 STEP 2453 H=2453:00 +CPU= 20.474
  16:07:35 STEP 2454 H=2454:00 +CPU= 37.089
  16:07:55 STEP 2455 H=2455:00 +CPU= 19.911
  16:08:16 STEP 2456 H=2456:00 +CPU= 21.212
  16:08:50 STEP 2457 H=2457:00 +CPU= 33.394
  16:09:11 STEP 2458 H=2458:00 +CPU= 21.175
  16:09:31 STEP 2459 H=2459:00 +CPU= 20.001
  16:10:07 STEP 2460 H=2460:00 +CPU= 36.517
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002460
  16:10:30 STEP 2461 H=2461:00 +CPU= 22.933
  16:10:54 STEP 2462 H=2462:00 +CPU= 23.457
  16:11:35 STEP 2463 H=2463:00 +CPU= 41.079
  16:11:58 STEP 2464 H=2464:00 +CPU= 23.477
  16:12:21 STEP 2465 H=2465:00 +CPU= 22.768
  16:13:06 STEP 2466 H=2466:00 +CPU= 44.314
  16:13:31 STEP 2467 H=2467:00 +CPU= 25.378
  16:13:52 STEP 2468 H=2468:00 +CPU= 21.436
  16:14:27 STEP 2469 H=2469:00 +CPU= 34.220
  16:14:49 STEP 2470 H=2470:00 +CPU= 22.570
  16:15:14 STEP 2471 H=2471:00 +CPU= 24.794
  16:15:52 STEP 2472 H=2472:00 +CPU= 38.429
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_102.zip
Uploading the intermediate file: upload_file_102.zip
  16:16:13 STEP 2473 H=2473:00 +CPU= 20.692
  16:16:38 STEP 2474 H=2474:00 +CPU= 24.388
  16:17:15 STEP 2475 H=2475:00 +CPU= 37.665
  16:17:38 STEP 2476 H=2476:00 +CPU= 23.078
  16:18:00 STEP 2477 H=2477:00 +CPU= 21.529
  16:18:36 STEP 2478 H=2478:00 +CPU= 36.187
  16:18:56 STEP 2479 H=2479:00 +CPU= 19.700
  16:19:18 STEP 2480 H=2480:00 +CPU= 22.614
  16:19:50 STEP 2481 H=2481:00 +CPU= 32.043
  16:20:11 STEP 2482 H=2482:00 +CPU= 20.930
  16:20:32 STEP 2483 H=2483:00 +CPU= 20.485
  16:21:09 STEP 2484 H=2484:00 +CPU= 37.201
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002484
  16:21:30 STEP 2485 H=2485:00 +CPU= 20.580
  16:21:53 STEP 2486 H=2486:00 +CPU= 22.913
  16:22:28 STEP 2487 H=2487:00 +CPU= 35.777
  16:22:54 STEP 2488 H=2488:00 +CPU= 25.543
  16:23:15 STEP 2489 H=2489:00 +CPU= 20.648
  16:23:49 STEP 2490 H=2490:00 +CPU= 34.277
  16:24:14 STEP 2491 H=2491:00 +CPU= 25.274
  16:24:40 STEP 2492 H=2492:00 +CPU= 25.791
  16:25:18 STEP 2493 H=2493:00 +CPU= 38.348
  16:25:42 STEP 2494 H=2494:00 +CPU= 23.516
  16:26:06 STEP 2495 H=2495:00 +CPU= 23.589
  16:26:51 STEP 2496 H=2496:00 +CPU= 45.212
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_103.zip
Uploading the intermediate file: upload_file_103.zip
  16:27:13 STEP 2497 H=2497:00 +CPU= 22.136
  16:27:33 STEP 2498 H=2498:00 +CPU= 20.406
  16:28:04 STEP 2499 H=2499:00 +CPU= 31.130
  16:28:26 STEP 2500 H=2500:00 +CPU= 21.781
  16:28:46 STEP 2501 H=2501:00 +CPU= 19.578
  16:29:19 STEP 2502 H=2502:00 +CPU= 33.047
  16:29:40 STEP 2503 H=2503:00 +CPU= 20.969
  16:30:02 STEP 2504 H=2504:00 +CPU= 22.010
  16:30:34 STEP 2505 H=2505:00 +CPU= 32.082
  16:30:54 STEP 2506 H=2506:00 +CPU= 20.205
  16:31:15 STEP 2507 H=2507:00 +CPU= 21.023
  16:32:01 STEP 2508 H=2508:00 +CPU= 45.889
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002508
  16:32:22 STEP 2509 H=2509:00 +CPU= 21.042
  16:32:49 STEP 2510 H=2510:00 +CPU= 26.485
  16:33:21 STEP 2511 H=2511:00 +CPU= 32.797
  16:33:44 STEP 2512 H=2512:00 +CPU= 22.226
  16:34:07 STEP 2513 H=2513:00 +CPU= 23.302
  16:34:42 STEP 2514 H=2514:00 +CPU= 34.954
  16:35:06 STEP 2515 H=2515:00 +CPU= 23.662
  16:35:28 STEP 2516 H=2516:00 +CPU= 22.129
  16:36:05 STEP 2517 H=2517:00 +CPU= 37.009
  16:36:32 STEP 2518 H=2518:00 +CPU= 26.895
  16:36:54 STEP 2519 H=2519:00 +CPU= 22.827
  16:37:41 STEP 2520 H=2520:00 +CPU= 46.433
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_104.zip
Uploading the intermediate file: upload_file_104.zip
  16:38:06 STEP 2521 H=2521:00 +CPU= 25.396
  16:38:31 STEP 2522 H=2522:00 +CPU= 24.426
  16:39:08 STEP 2523 H=2523:00 +CPU= 37.475
  16:39:32 STEP 2524 H=2524:00 +CPU= 23.899
  16:39:54 STEP 2525 H=2525:00 +CPU= 22.015
  16:40:30 STEP 2526 H=2526:00 +CPU= 35.895
  16:40:52 STEP 2527 H=2527:00 +CPU= 21.828
  16:41:17 STEP 2528 H=2528:00 +CPU= 25.091
  16:41:49 STEP 2529 H=2529:00 +CPU= 32.464
  16:42:11 STEP 2530 H=2530:00 +CPU= 21.749
  16:42:31 STEP 2531 H=2531:00 +CPU= 19.858
  16:43:10 STEP 2532 H=2532:00 +CPU= 38.870
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002532
  16:43:33 STEP 2533 H=2533:00 +CPU= 23.181
  16:43:55 STEP 2534 H=2534:00 +CPU= 22.254
  16:44:31 STEP 2535 H=2535:00 +CPU= 35.671
  16:44:53 STEP 2536 H=2536:00 +CPU= 22.368
  16:45:16 STEP 2537 H=2537:00 +CPU= 22.348
  16:45:51 STEP 2538 H=2538:00 +CPU= 35.222
  16:46:16 STEP 2539 H=2539:00 +CPU= 24.821
  16:46:43 STEP 2540 H=2540:00 +CPU= 26.792
  16:47:20 STEP 2541 H=2541:00 +CPU= 37.022
  16:47:44 STEP 2542 H=2542:00 +CPU= 24.454
  16:48:05 STEP 2543 H=2543:00 +CPU= 20.522
  16:48:51 STEP 2544 H=2544:00 +CPU= 46.759
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_105.zip
Uploading the intermediate file: upload_file_105.zip
  16:49:19 STEP 2545 H=2545:00 +CPU= 27.844
  16:49:46 STEP 2546 H=2546:00 +CPU= 26.324
  16:50:19 STEP 2547 H=2547:00 +CPU= 32.925
  16:50:43 STEP 2548 H=2548:00 +CPU= 24.304
  16:51:07 STEP 2549 H=2549:00 +CPU= 24.108
  16:51:47 STEP 2550 H=2550:00 +CPU= 40.375
  16:52:14 STEP 2551 H=2551:00 +CPU= 26.291
  16:52:36 STEP 2552 H=2552:00 +CPU= 22.111
  16:53:08 STEP 2553 H=2553:00 +CPU= 32.481
  16:53:33 STEP 2554 H=2554:00 +CPU= 24.654
  16:54:01 STEP 2555 H=2555:00 +CPU= 27.570
  16:54:44 STEP 2556 H=2556:00 +CPU= 43.696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002556
  16:55:05 STEP 2557 H=2557:00 +CPU= 20.329
  16:55:29 STEP 2558 H=2558:00 +CPU= 24.188
  16:56:06 STEP 2559 H=2559:00 +CPU= 37.180
  16:56:31 STEP 2560 H=2560:00 +CPU= 25.001
  16:56:58 STEP 2561 H=2561:00 +CPU= 26.784
  16:57:37 STEP 2562 H=2562:00 +CPU= 39.433
  16:58:03 STEP 2563 H=2563:00 +CPU= 25.498
  16:58:25 STEP 2564 H=2564:00 +CPU= 22.633
  16:59:10 STEP 2565 H=2565:00 +CPU= 44.461
  16:59:37 STEP 2566 H=2566:00 +CPU= 27.339
  16:59:58 STEP 2567 H=2567:00 +CPU= 20.275
  17:00:47 STEP 2568 H=2568:00 +CPU= 49.203
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_106.zip
Uploading the intermediate file: upload_file_106.zip
  17:01:07 STEP 2569 H=2569:00 +CPU= 19.847
  17:01:27 STEP 2570 H=2570:00 +CPU= 20.616
  17:02:08 STEP 2571 H=2571:00 +CPU= 40.252
  17:02:29 STEP 2572 H=2572:00 +CPU= 21.755
  17:02:57 STEP 2573 H=2573:00 +CPU= 27.538
  17:03:40 STEP 2574 H=2574:00 +CPU= 42.689
  17:04:08 STEP 2575 H=2575:00 +CPU= 27.891
  17:04:33 STEP 2576 H=2576:00 +CPU= 25.198
  17:05:09 STEP 2577 H=2577:00 +CPU= 35.666
  17:05:29 STEP 2578 H=2578:00 +CPU= 20.689
  17:05:56 STEP 2579 H=2579:00 +CPU= 26.396
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4793MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4793 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170626:1673046386:17592.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170627:1673046387:17593.483] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42063:42063] [20230106:170627:1673046387:17593.483] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC0A80  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000101C97B  cloudsc_                  857  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F10C1B  callpar_                 1011  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2014050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0845
(argv4) batchid: 983
(argv5) wuid: 12200489
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b664959745c7ad5f1d0a8e57ec1a940a to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0845_2014050100_123_983_12200489.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0845_2014050100_123_983_12200489.zip
ic_ancil_file: ic_ancil_12200489
ifsdata_file: ifsdata_12200489
climate_data_file: clim_data_12200489
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8b756c0108802b530e2bd8acf613d51d to: /var/lib/boinc-client/slots/1/ic_ancil_12200489.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12200489.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_42e6adaf297b0347ac0bbd793288a49e to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200489.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12200489.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200489.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12200489.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12200489.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12200489.xml
-- Model is restarting --
Adjusting last_iter, 2580, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12200489.xml
last_cpu_time: 77177
upload_file_number: 107
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106
The child process has been launched with process id: 42934
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-03.fullford.com:1:1:42934:42934] [20230106:172614:1673047574:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:26:55 STEP 2568 H=2568:00 +CPU= 40.008
  17:27:17 STEP 2569 H=2569:00 +CPU= 21.776
  17:27:39 STEP 2570 H=2570:00 +CPU= 22.021
  17:28:10 STEP 2571 H=2571:00 +CPU= 31.035
  17:28:31 STEP 2572 H=2572:00 +CPU= 20.558
  17:28:50 STEP 2573 H=2573:00 +CPU= 18.636
  17:29:23 STEP 2574 H=2574:00 +CPU= 33.885
  17:29:43 STEP 2575 H=2575:00 +CPU= 19.542
  17:30:07 STEP 2576 H=2576:00 +CPU= 23.876
  17:30:48 STEP 2577 H=2577:00 +CPU= 40.935
  17:31:15 STEP 2578 H=2578:00 +CPU= 27.637
  17:31:42 STEP 2579 H=2579:00 +CPU= 26.500
  17:32:26 STEP 2580 H=2580:00 +CPU= 44.323
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002580
  17:32:49 STEP 2581 H=2581:00 +CPU= 22.376
  17:33:12 STEP 2582 H=2582:00 +CPU= 22.980
  17:33:48 STEP 2583 H=2583:00 +CPU= 36.099
  17:34:12 STEP 2584 H=2584:00 +CPU= 23.618
  17:34:31 STEP 2585 H=2585:00 +CPU= 19.940
  17:35:04 STEP 2586 H=2586:00 +CPU= 32.268
  17:35:23 STEP 2587 H=2587:00 +CPU= 18.939
  17:35:44 STEP 2588 H=2588:00 +CPU= 21.502
  17:36:15 STEP 2589 H=2589:00 +CPU= 31.288
  17:36:38 STEP 2590 H=2590:00 +CPU= 22.326
  17:37:00 STEP 2591 H=2591:00 +CPU= 22.657
  17:37:42 STEP 2592 H=2592:00 +CPU= 41.718
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12200489/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0845_2014050100_123_983_12200489_0_r1005712106_107.zip
Uploading the intermediate file: upload_file_107.zip
  17:38:04 STEP 2593 H=2593:00 +CPU= 22.109
  17:38:30 STEP 2594 H=2594:00 +CPU= 26.031
  17:39:02 STEP 2595 H=2595:00 +CPU= 31.305
  17:39:22 STEP 2596 H=2596:00 +CPU= 20.221
  17:39:43 STEP 2597 H=2597:00 +CPU= 21.184
  17:40:15 STEP 2598 H=2598:00 +CPU= 32.317
  17:40:37 STEP 2599 H=2599:00 +CPU= 21.706
  17:41:01 STEP 2600 H=2600:00 +CPU= 23.893
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:10:58 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 8,726,400 72,856 0.0083
09 May 2023 16:10:58 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 8,726,400 72,856 0.0083
09 May 2023 16:10:51 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 7,862,400 66,461 0.0085
09 May 2023 16:10:16 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 4,752,000 38,801 0.0082
09 May 2023 16:10:13 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 3,888,000 31,995 0.0082
09 May 2023 16:10:03 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 3,024,000 25,259 0.0084
09 May 2023 16:09:59 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 2,160,000 18,545 0.0086
09 May 2023 16:09:52 1459164 22286718 oifs_43r3_ps_0845_2014050100_123_983_12200489_0 1,296,000 11,349 0.0088


©2024 climateprediction.net