climateprediction.net home page
Task 22275783

Task 22275783

Name oifs_43r3_ps_0553_2004050100_123_973_12190197_0
Workunit 12190197
Created 23 Dec 2022, 8:15:53 UTC
Sent 25 Dec 2022, 9:49:14 UTC
Report deadline 24 Jan 2023, 9:49:14 UTC
Received 15 Feb 2023, 20:28:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1495246
Run time 14 hours 53 min 25 sec
CPU time 14 hours 34 min 38 sec
Validate state Invalid
Credit 12,216.96
Device peak FLOPS 4.30 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,799.48 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.62 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
tment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230128:125731:1674907051:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:58:17 STEP 2184 H=2184:00 +CPU= 27.521
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:59:29 STEP 2185 H=2185:00 +CPU= 18.142
  12:59:44 STEP 2186 H=2186:00 +CPU= 14.985
  13:00:10 STEP 2187 H=2187:00 +CPU= 24.290
  13:00:27 STEP 2188 H=2188:00 +CPU= 16.579
  13:00:45 STEP 2189 H=2189:00 +CPU= 17.010
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:01:22 STEP 2190 H=2190:00 +CPU= 24.887
  13:01:37 STEP 2191 H=2191:00 +CPU= 15.030
  13:01:53 STEP 2192 H=2192:00 +CPU= 16.017
  13:02:19 STEP 2193 H=2193:00 +CPU= 24.752
  13:02:36 STEP 2194 H=2194:00 +CPU= 16.183
  13:02:52 STEP 2195 H=2195:00 +CPU= 15.718
  13:03:19 STEP 2196 H=2196:00 +CPU= 26.785
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002196
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002196
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002196
  13:03:35 STEP 2197 H=2197:00 +CPU= 15.571
  13:03:51 STEP 2198 H=2198:00 +CPU= 15.997
  13:04:15 STEP 2199 H=2199:00 +CPU= 23.889
  13:04:31 STEP 2200 H=2200:00 +CPU= 15.696
  13:04:47 STEP 2201 H=2201:00 +CPU= 15.431
  13:05:12 STEP 2202 H=2202:00 +CPU= 24.457
  13:05:27 STEP 2203 H=2203:00 +CPU= 15.376
  13:05:43 STEP 2204 H=2204:00 +CPU= 15.940
  13:06:08 STEP 2205 H=2205:00 +CPU= 23.862
  13:06:24 STEP 2206 H=2206:00 +CPU= 15.898
  13:06:40 STEP 2207 H=2207:00 +CPU= 15.947
  13:07:09 STEP 2208 H=2208:00 +CPU= 28.102
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002208
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002208
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002208
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002184
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002184
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002184
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002196
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002196
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002196
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_91.zip
Uploading the intermediate file: upload_file_91.zip
  13:07:25 STEP 2209 H=2209:00 +CPU= 15.879
  13:07:41 STEP 2210 H=2210:00 +CPU= 15.835
  13:08:07 STEP 2211 H=2211:00 +CPU= 24.765
  13:08:23 STEP 2212 H=2212:00 +CPU= 16.035
  13:08:39 STEP 2213 H=2213:00 +CPU= 15.593
  13:09:05 STEP 2214 H=2214:00 +CPU= 24.892
  13:09:20 STEP 2215 H=2215:00 +CPU= 15.534
  13:09:37 STEP 2216 H=2216:00 +CPU= 16.301
  13:10:03 STEP 2217 H=2217:00 +CPU= 24.504
  13:10:19 STEP 2218 H=2218:00 +CPU= 15.992
  13:10:36 STEP 2219 H=2219:00 +CPU= 16.444
  13:11:07 STEP 2220 H=2220:00 +CPU= 28.719
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002220
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002220
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002220
  13:11:23 STEP 2221 H=2221:00 +CPU= 15.813
  13:11:41 STEP 2222 H=2222:00 +CPU= 16.970
  13:12:07 STEP 2223 H=2223:00 +CPU= 24.896
  13:12:23 STEP 2224 H=2224:00 +CPU= 16.252
  13:12:40 STEP 2225 H=2225:00 +CPU= 16.238
  13:13:08 STEP 2226 H=2226:00 +CPU= 26.854
  13:13:24 STEP 2227 H=2227:00 +CPU= 15.875
  13:13:41 STEP 2228 H=2228:00 +CPU= 16.178
  13:14:05 STEP 2229 H=2229:00 +CPU= 24.483
  13:14:21 STEP 2230 H=2230:00 +CPU= 15.811
  13:14:37 STEP 2231 H=2231:00 +CPU= 16.118
  13:15:05 STEP 2232 H=2232:00 +CPU= 27.805
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002232
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002232
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002232
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002208
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002208
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002208
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002220
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002220
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002220
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_92.zip
Uploading the intermediate file: upload_file_92.zip
  13:15:22 STEP 2233 H=2233:00 +CPU= 15.899
  13:15:38 STEP 2234 H=2234:00 +CPU= 16.074
  13:16:02 STEP 2235 H=2235:00 +CPU= 23.999
  13:16:18 STEP 2236 H=2236:00 +CPU= 15.826
  13:16:33 STEP 2237 H=2237:00 +CPU= 15.423
  13:16:58 STEP 2238 H=2238:00 +CPU= 24.444
  13:17:14 STEP 2239 H=2239:00 +CPU= 15.498
  13:17:29 STEP 2240 H=2240:00 +CPU= 15.662
  13:17:53 STEP 2241 H=2241:00 +CPU= 23.820
  13:18:10 STEP 2242 H=2242:00 +CPU= 16.265
  13:18:25 STEP 2243 H=2243:00 +CPU= 15.475
  13:18:52 STEP 2244 H=2244:00 +CPU= 26.554
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002244
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002244
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002244
  13:19:08 STEP 2245 H=2245:00 +CPU= 15.457
  13:19:24 STEP 2246 H=2246:00 +CPU= 16.044
  13:19:48 STEP 2247 H=2247:00 +CPU= 23.631
  13:20:03 STEP 2248 H=2248:00 +CPU= 15.725
  13:20:19 STEP 2249 H=2249:00 +CPU= 15.412
  13:20:44 STEP 2250 H=2250:00 +CPU= 24.737
  13:20:59 STEP 2251 H=2251:00 +CPU= 15.432
  13:21:15 STEP 2252 H=2252:00 +CPU= 15.955
  13:21:39 STEP 2253 H=2253:00 +CPU= 23.883
  13:21:55 STEP 2254 H=2254:00 +CPU= 15.634
  13:22:11 STEP 2255 H=2255:00 +CPU= 15.806
  13:22:39 STEP 2256 H=2256:00 +CPU= 27.674
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002256
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002256
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002256
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002232
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002232
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002232
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002244
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002244
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002244
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_93.zip
Uploading the intermediate file: upload_file_93.zip
  13:22:55 STEP 2257 H=2257:00 +CPU= 15.857
  13:23:11 STEP 2258 H=2258:00 +CPU= 15.987
  13:23:35 STEP 2259 H=2259:00 +CPU= 23.761
  13:23:51 STEP 2260 H=2260:00 +CPU= 15.739
  13:24:06 STEP 2261 H=2261:00 +CPU= 15.416
  13:24:31 STEP 2262 H=2262:00 +CPU= 24.375
  13:24:46 STEP 2263 H=2263:00 +CPU= 15.222
  13:25:02 STEP 2264 H=2264:00 +CPU= 15.899
  13:25:26 STEP 2265 H=2265:00 +CPU= 23.936
  13:25:42 STEP 2266 H=2266:00 +CPU= 15.935
  13:25:58 STEP 2267 H=2267:00 +CPU= 15.616
  13:26:25 STEP 2268 H=2268:00 +CPU= 26.733
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002268
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002268
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002268
  13:26:40 STEP 2269 H=2269:00 +CPU= 15.417
  13:26:56 STEP 2270 H=2270:00 +CPU= 15.960
  13:27:21 STEP 2271 H=2271:00 +CPU= 24.305
  13:27:37 STEP 2272 H=2272:00 +CPU= 15.829
  13:27:52 STEP 2273 H=2273:00 +CPU= 15.323
  13:28:17 STEP 2274 H=2274:00 +CPU= 24.693
  13:28:32 STEP 2275 H=2275:00 +CPU= 15.476
  13:28:49 STEP 2276 H=2276:00 +CPU= 15.896
  13:29:13 STEP 2277 H=2277:00 +CPU= 23.918
  13:29:29 STEP 2278 H=2278:00 +CPU= 15.970
  13:29:45 STEP 2279 H=2279:00 +CPU= 15.912
  13:30:13 STEP 2280 H=2280:00 +CPU= 28.419
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002280
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002280
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002280
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002256
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002256
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002256
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002268
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002268
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002268
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_94.zip
Uploading the intermediate file: upload_file_94.zip
  13:30:29 STEP 2281 H=2281:00 +CPU= 15.670
  13:30:46 STEP 2282 H=2282:00 +CPU= 16.210
  13:31:10 STEP 2283 H=2283:00 +CPU= 24.012
  13:31:26 STEP 2284 H=2284:00 +CPU= 16.004
  13:31:42 STEP 2285 H=2285:00 +CPU= 15.306
  13:32:06 STEP 2286 H=2286:00 +CPU= 24.751
  13:32:22 STEP 2287 H=2287:00 +CPU= 15.311
  13:32:38 STEP 2288 H=2288:00 +CPU= 15.984
  13:33:02 STEP 2289 H=2289:00 +CPU= 24.147
  13:33:18 STEP 2290 H=2290:00 +CPU= 15.808
  13:33:34 STEP 2291 H=2291:00 +CPU= 15.323
  13:34:00 STEP 2292 H=2292:00 +CPU= 26.161
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002292
  13:34:16 STEP 2293 H=2293:00 +CPU= 15.400
  13:34:32 STEP 2294 H=2294:00 +CPU= 15.891
  13:34:56 STEP 2295 H=2295:00 +CPU= 23.918
  13:35:12 STEP 2296 H=2296:00 +CPU= 16.060
  13:35:28 STEP 2297 H=2297:00 +CPU= 15.806
  13:35:52 STEP 2298 H=2298:00 +CPU= 24.331
  13:36:08 STEP 2299 H=2299:00 +CPU= 15.619
  13:36:24 STEP 2300 H=2300:00 +CPU= 16.045
  13:36:48 STEP 2301 H=2301:00 +CPU= 23.910
  13:37:04 STEP 2302 H=2302:00 +CPU= 15.827
  13:37:20 STEP 2303 H=2303:00 +CPU= 15.751
  13:37:48 STEP 2304 H=2304:00 +CPU= 27.857
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002280
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002280
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002280
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002292
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002292
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002292
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_95.zip
Uploading the intermediate file: upload_file_95.zip
  13:38:04 STEP 2305 H=2305:00 +CPU= 15.792
  13:38:21 STEP 2306 H=2306:00 +CPU= 16.506
  13:38:45 STEP 2307 H=2307:00 +CPU= 23.866
  13:39:01 STEP 2308 H=2308:00 +CPU= 15.809
  13:39:16 STEP 2309 H=2309:00 +CPU= 15.474
  13:39:41 STEP 2310 H=2310:00 +CPU= 24.297
  13:39:56 STEP 2311 H=2311:00 +CPU= 15.543
  13:40:12 STEP 2312 H=2312:00 +CPU= 15.777
  13:40:36 STEP 2313 H=2313:00 +CPU= 23.908
  13:40:52 STEP 2314 H=2314:00 +CPU= 16.100
  13:41:08 STEP 2315 H=2315:00 +CPU= 15.632
  13:41:35 STEP 2316 H=2316:00 +CPU= 26.807
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002316
  13:41:50 STEP 2317 H=2317:00 +CPU= 15.209
  13:42:06 STEP 2318 H=2318:00 +CPU= 15.861
  13:42:31 STEP 2319 H=2319:00 +CPU= 23.906
  13:42:47 STEP 2320 H=2320:00 +CPU= 15.988
  13:43:02 STEP 2321 H=2321:00 +CPU= 15.457
  13:43:27 STEP 2322 H=2322:00 +CPU= 24.261
  13:43:42 STEP 2323 H=2323:00 +CPU= 15.494
  13:44:00 STEP 2324 H=2324:00 +CPU= 17.881
  13:44:25 STEP 2325 H=2325:00 +CPU= 24.412
  13:44:42 STEP 2326 H=2326:00 +CPU= 15.898
  13:44:58 STEP 2327 H=2327:00 +CPU= 16.182
  13:45:26 STEP 2328 H=2328:00 +CPU= 27.958
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002304
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002304
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002304
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002316
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002316
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002316
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_96.zip
Uploading the intermediate file: upload_file_96.zip
  13:45:42 STEP 2329 H=2329:00 +CPU= 15.583
  13:45:58 STEP 2330 H=2330:00 +CPU= 16.102
  13:46:22 STEP 2331 H=2331:00 +CPU= 23.974
  13:46:38 STEP 2332 H=2332:00 +CPU= 16.108
  13:46:54 STEP 2333 H=2333:00 +CPU= 15.315
  13:47:18 STEP 2334 H=2334:00 +CPU= 24.339
  13:47:34 STEP 2335 H=2335:00 +CPU= 15.504
  13:47:50 STEP 2336 H=2336:00 +CPU= 16.080
  13:48:14 STEP 2337 H=2337:00 +CPU= 24.102
  13:48:30 STEP 2338 H=2338:00 +CPU= 15.866
  13:48:46 STEP 2339 H=2339:00 +CPU= 15.254
  13:49:12 STEP 2340 H=2340:00 +CPU= 26.506
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002340
  13:49:28 STEP 2341 H=2341:00 +CPU= 15.277
  13:49:44 STEP 2342 H=2342:00 +CPU= 15.939
  13:50:08 STEP 2343 H=2343:00 +CPU= 23.642
  13:50:24 STEP 2344 H=2344:00 +CPU= 15.569
  13:50:39 STEP 2345 H=2345:00 +CPU= 15.454
  13:51:04 STEP 2346 H=2346:00 +CPU= 24.638
  13:51:19 STEP 2347 H=2347:00 +CPU= 15.321
  13:51:35 STEP 2348 H=2348:00 +CPU= 15.805
  13:51:59 STEP 2349 H=2349:00 +CPU= 23.830
  13:52:15 STEP 2350 H=2350:00 +CPU= 15.762
  13:52:31 STEP 2351 H=2351:00 +CPU= 15.838
  13:52:59 STEP 2352 H=2352:00 +CPU= 28.265
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002328
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002328
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002328
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002340
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002340
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002340
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_97.zip
Uploading the intermediate file: upload_file_97.zip
  13:53:15 STEP 2353 H=2353:00 +CPU= 15.884
  13:53:31 STEP 2354 H=2354:00 +CPU= 15.964
  13:53:55 STEP 2355 H=2355:00 +CPU= 23.653
  13:54:11 STEP 2356 H=2356:00 +CPU= 15.780
  13:54:27 STEP 2357 H=2357:00 +CPU= 15.375
  13:54:51 STEP 2358 H=2358:00 +CPU= 24.455
  13:55:07 STEP 2359 H=2359:00 +CPU= 15.422
  13:55:23 STEP 2360 H=2360:00 +CPU= 16.172
  13:55:48 STEP 2361 H=2361:00 +CPU= 24.011
  13:56:03 STEP 2362 H=2362:00 +CPU= 15.748
  13:56:19 STEP 2363 H=2363:00 +CPU= 15.422
  13:56:45 STEP 2364 H=2364:00 +CPU= 26.183
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002364
  13:57:01 STEP 2365 H=2365:00 +CPU= 15.347
  13:57:17 STEP 2366 H=2366:00 +CPU= 16.155
  13:57:41 STEP 2367 H=2367:00 +CPU= 23.892
  13:57:58 STEP 2368 H=2368:00 +CPU= 16.658
  13:58:13 STEP 2369 H=2369:00 +CPU= 15.503
  13:58:38 STEP 2370 H=2370:00 +CPU= 24.489
  13:58:53 STEP 2371 H=2371:00 +CPU= 15.368
  13:59:09 STEP 2372 H=2372:00 +CPU= 16.102
  13:59:34 STEP 2373 H=2373:00 +CPU= 23.952
  13:59:49 STEP 2374 H=2374:00 +CPU= 15.821
  14:00:06 STEP 2375 H=2375:00 +CPU= 16.110
  14:00:34 STEP 2376 H=2376:00 +CPU= 27.800
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002352
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002352
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002352
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002364
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002364
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002364
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_98.zip
Uploading the intermediate file: upload_file_98.zip
  14:00:49 STEP 2377 H=2377:00 +CPU= 15.495
  14:01:06 STEP 2378 H=2378:00 +CPU= 15.898
  14:01:30 STEP 2379 H=2379:00 +CPU= 23.691
  14:01:46 STEP 2380 H=2380:00 +CPU= 15.958
  14:02:01 STEP 2381 H=2381:00 +CPU= 15.477
  14:02:26 STEP 2382 H=2382:00 +CPU= 24.305
  14:02:41 STEP 2383 H=2383:00 +CPU= 15.119
  14:02:57 STEP 2384 H=2384:00 +CPU= 16.191
  14:03:21 STEP 2385 H=2385:00 +CPU= 24.021
  14:03:37 STEP 2386 H=2386:00 +CPU= 15.888
  14:03:53 STEP 2387 H=2387:00 +CPU= 15.523
  14:04:20 STEP 2388 H=2388:00 +CPU= 26.620
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002388
  14:04:35 STEP 2389 H=2389:00 +CPU= 15.374
  14:04:51 STEP 2390 H=2390:00 +CPU= 15.693
  14:05:15 STEP 2391 H=2391:00 +CPU= 23.739
  14:05:31 STEP 2392 H=2392:00 +CPU= 15.961
  14:05:46 STEP 2393 H=2393:00 +CPU= 15.315
  14:06:11 STEP 2394 H=2394:00 +CPU= 24.737
  14:06:27 STEP 2395 H=2395:00 +CPU= 15.329
  14:06:43 STEP 2396 H=2396:00 +CPU= 15.996
  14:07:07 STEP 2397 H=2397:00 +CPU= 23.647
  14:07:23 STEP 2398 H=2398:00 +CPU= 15.800
  14:07:39 STEP 2399 H=2399:00 +CPU= 16.138
  14:08:08 STEP 2400 H=2400:00 +CPU= 28.760
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002376
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002376
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002376
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002388
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002388
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002388
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_99.zip
Uploading the intermediate file: upload_file_99.zip
  19:15:39 STEP 2401 H=2401:00 +CPU= 16.955
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:16:21 STEP 2402 H=2402:00 +CPU= 18.886
  19:16:47 STEP 2403 H=2403:00 +CPU= 24.935
  19:17:04 STEP 2404 H=2404:00 +CPU= 16.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:17:43 STEP 2405 H=2405:00 +CPU= 16.836
  19:18:11 STEP 2406 H=2406:00 +CPU= 26.086
  19:18:29 STEP 2407 H=2407:00 +CPU= 16.883
  19:18:44 STEP 2408 H=2408:00 +CPU= 15.631
  19:19:11 STEP 2409 H=2409:00 +CPU= 25.668
  19:19:29 STEP 2410 H=2410:00 +CPU= 17.120
  19:19:46 STEP 2411 H=2411:00 +CPU= 16.112
  19:20:12 STEP 2412 H=2412:00 +CPU= 26.089
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002412
  19:20:28 STEP 2413 H=2413:00 +CPU= 15.988
  19:20:45 STEP 2414 H=2414:00 +CPU= 16.069
  19:21:08 STEP 2415 H=2415:00 +CPU= 23.534
  19:21:24 STEP 2416 H=2416:00 +CPU= 15.384
  19:21:39 STEP 2417 H=2417:00 +CPU= 14.950
  19:22:03 STEP 2418 H=2418:00 +CPU= 24.076
  19:22:18 STEP 2419 H=2419:00 +CPU= 15.121
  19:22:34 STEP 2420 H=2420:00 +CPU= 15.637
  19:22:58 STEP 2421 H=2421:00 +CPU= 23.300
  19:23:14 STEP 2422 H=2422:00 +CPU= 15.822
  19:23:30 STEP 2423 H=2423:00 +CPU= 15.857
  19:23:57 STEP 2424 H=2424:00 +CPU= 27.646
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002400
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002400
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002400
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002412
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002412
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002412
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_100.zip
Uploading the intermediate file: upload_file_100.zip
  19:24:13 STEP 2425 H=2425:00 +CPU= 15.461
Uploading trickle at timestep: 8726400
  19:24:29 STEP 2426 H=2426:00 +CPU= 15.681
  19:24:53 STEP 2427 H=2427:00 +CPU= 23.602
  19:25:08 STEP 2428 H=2428:00 +CPU= 15.479
  19:25:23 STEP 2429 H=2429:00 +CPU= 14.936
  19:25:47 STEP 2430 H=2430:00 +CPU= 24.018
  19:26:03 STEP 2431 H=2431:00 +CPU= 15.260
  19:26:18 STEP 2432 H=2432:00 +CPU= 15.598
  18:51:01 STEP 2433 H=2433:00 +CPU= 23.687
  18:51:18 STEP 2434 H=2434:00 +CPU= 16.473
  18:51:35 STEP 2435 H=2435:00 +CPU= 16.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:52:37 STEP 2436 H=2436:00 +CPU= 28.469
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002436
  18:52:53 STEP 2437 H=2437:00 +CPU= 15.769
  18:53:10 STEP 2438 H=2438:00 +CPU= 16.225
  18:53:34 STEP 2439 H=2439:00 +CPU= 24.125
  18:53:51 STEP 2440 H=2440:00 +CPU= 16.174
  18:54:07 STEP 2441 H=2441:00 +CPU= 15.308
  18:54:33 STEP 2442 H=2442:00 +CPU= 25.479
  18:54:50 STEP 2443 H=2443:00 +CPU= 15.934
  18:55:07 STEP 2444 H=2444:00 +CPU= 16.667
  18:55:34 STEP 2445 H=2445:00 +CPU= 25.672
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:56:03 STEP 2446 H=2446:00 +CPU= 16.546
  18:56:18 STEP 2447 H=2447:00 +CPU= 15.237
  18:56:47 STEP 2448 H=2448:00 +CPU= 28.550
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002424
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002424
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002424
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002436
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002436
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002436
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_101.zip
Uploading the intermediate file: upload_file_101.zip
  18:57:03 STEP 2449 H=2449:00 +CPU= 15.946
  18:57:20 STEP 2450 H=2450:00 +CPU= 15.931
  18:57:44 STEP 2451 H=2451:00 +CPU= 24.136
  18:58:01 STEP 2452 H=2452:00 +CPU= 15.895
  18:58:16 STEP 2453 H=2453:00 +CPU= 15.437
  18:58:41 STEP 2454 H=2454:00 +CPU= 24.519
  18:58:57 STEP 2455 H=2455:00 +CPU= 15.476
  18:59:14 STEP 2456 H=2456:00 +CPU= 16.918
  18:59:39 STEP 2457 H=2457:00 +CPU= 24.137
  18:59:56 STEP 2458 H=2458:00 +CPU= 16.327
  19:00:13 STEP 2459 H=2459:00 +CPU= 16.571
  19:00:42 STEP 2460 H=2460:00 +CPU= 27.793
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002460
  19:00:59 STEP 2461 H=2461:00 +CPU= 16.535
  19:01:18 STEP 2462 H=2462:00 +CPU= 17.717
  19:01:45 STEP 2463 H=2463:00 +CPU= 26.004
  19:02:02 STEP 2464 H=2464:00 +CPU= 16.414
  19:02:22 STEP 2465 H=2465:00 +CPU= 18.299
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  19:02:58 STEP 2466 H=2466:00 +CPU= 24.195
  19:03:14 STEP 2467 H=2467:00 +CPU= 16.049
  19:03:30 STEP 2468 H=2468:00 +CPU= 15.700
  19:03:56 STEP 2469 H=2469:00 +CPU= 24.957
  19:04:12 STEP 2470 H=2470:00 +CPU= 15.892
  19:04:28 STEP 2471 H=2471:00 +CPU= 16.124
  19:04:57 STEP 2472 H=2472:00 +CPU= 28.541
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002448
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002448
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002448
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002460
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002460
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002460
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_102.zip
Uploading the intermediate file: upload_file_102.zip
  19:05:14 STEP 2473 H=2473:00 +CPU= 15.732
  19:05:30 STEP 2474 H=2474:00 +CPU= 16.405
  19:05:54 STEP 2475 H=2475:00 +CPU= 23.805
  19:06:10 STEP 2476 H=2476:00 +CPU= 15.706
  19:06:26 STEP 2477 H=2477:00 +CPU= 15.466
  19:06:51 STEP 2478 H=2478:00 +CPU= 24.649
  19:07:06 STEP 2479 H=2479:00 +CPU= 15.418
  19:07:23 STEP 2480 H=2480:00 +CPU= 16.102
  19:07:47 STEP 2481 H=2481:00 +CPU= 24.047
  19:08:03 STEP 2482 H=2482:00 +CPU= 16.010
  19:08:18 STEP 2483 H=2483:00 +CPU= 15.256
  19:08:45 STEP 2484 H=2484:00 +CPU= 26.445
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002484
  19:09:01 STEP 2485 H=2485:00 +CPU= 15.500
  19:09:17 STEP 2486 H=2486:00 +CPU= 15.983
  19:09:41 STEP 2487 H=2487:00 +CPU= 23.777
  19:09:57 STEP 2488 H=2488:00 +CPU= 15.859
  19:10:12 STEP 2489 H=2489:00 +CPU= 15.294
  19:10:37 STEP 2490 H=2490:00 +CPU= 24.402
  19:10:53 STEP 2491 H=2491:00 +CPU= 15.550
  19:11:09 STEP 2492 H=2492:00 +CPU= 16.083
  19:11:33 STEP 2493 H=2493:00 +CPU= 23.606
  19:11:50 STEP 2494 H=2494:00 +CPU= 16.764
  19:12:06 STEP 2495 H=2495:00 +CPU= 16.073
  19:12:35 STEP 2496 H=2496:00 +CPU= 28.234
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002472
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002472
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002472
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002484
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002484
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002484
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_103.zip
Uploading the intermediate file: upload_file_103.zip
  19:12:51 STEP 2497 H=2497:00 +CPU= 16.295
  19:13:08 STEP 2498 H=2498:00 +CPU= 16.201
  19:13:32 STEP 2499 H=2499:00 +CPU= 24.158
  19:13:48 STEP 2500 H=2500:00 +CPU= 15.871
  19:14:04 STEP 2501 H=2501:00 +CPU= 15.534
  19:14:28 STEP 2502 H=2502:00 +CPU= 24.392
  19:14:44 STEP 2503 H=2503:00 +CPU= 15.253
  19:15:00 STEP 2504 H=2504:00 +CPU= 15.774
  19:15:24 STEP 2505 H=2505:00 +CPU= 23.963
  19:15:40 STEP 2506 H=2506:00 +CPU= 15.740
  19:15:55 STEP 2507 H=2507:00 +CPU= 15.315
  19:16:22 STEP 2508 H=2508:00 +CPU= 26.376
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002508
  19:16:37 STEP 2509 H=2509:00 +CPU= 15.306
  19:16:53 STEP 2510 H=2510:00 +CPU= 15.913
  19:17:17 STEP 2511 H=2511:00 +CPU= 23.789
  19:17:33 STEP 2512 H=2512:00 +CPU= 15.817
  19:17:49 STEP 2513 H=2513:00 +CPU= 15.497
  19:18:14 STEP 2514 H=2514:00 +CPU= 24.767
  19:18:30 STEP 2515 H=2515:00 +CPU= 15.623
  19:18:47 STEP 2516 H=2516:00 +CPU= 16.496
  19:19:11 STEP 2517 H=2517:00 +CPU= 24.086
  19:19:27 STEP 2518 H=2518:00 +CPU= 15.838
  19:19:43 STEP 2519 H=2519:00 +CPU= 15.785
  19:20:11 STEP 2520 H=2520:00 +CPU= 28.079
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002496
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002496
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002496
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002508
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002508
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002508
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_104.zip
Uploading the intermediate file: upload_file_104.zip
  19:20:27 STEP 2521 H=2521:00 +CPU= 15.979
  19:20:43 STEP 2522 H=2522:00 +CPU= 16.031
  19:21:08 STEP 2523 H=2523:00 +CPU= 24.188
  19:21:24 STEP 2524 H=2524:00 +CPU= 15.934
  19:21:40 STEP 2525 H=2525:00 +CPU= 15.590
  19:22:05 STEP 2526 H=2526:00 +CPU= 24.707
  19:22:20 STEP 2527 H=2527:00 +CPU= 15.403
  19:22:36 STEP 2528 H=2528:00 +CPU= 15.933
  19:23:00 STEP 2529 H=2529:00 +CPU= 24.071
  19:23:16 STEP 2530 H=2530:00 +CPU= 16.003
  19:23:32 STEP 2531 H=2531:00 +CPU= 15.489
  19:23:59 STEP 2532 H=2532:00 +CPU= 26.728
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002532
  19:24:14 STEP 2533 H=2533:00 +CPU= 15.436
  19:24:30 STEP 2534 H=2534:00 +CPU= 15.946
  19:24:55 STEP 2535 H=2535:00 +CPU= 24.011
  19:25:11 STEP 2536 H=2536:00 +CPU= 16.031
  19:25:27 STEP 2537 H=2537:00 +CPU= 15.337
  19:25:51 STEP 2538 H=2538:00 +CPU= 24.370
  19:26:07 STEP 2539 H=2539:00 +CPU= 15.462
  19:26:23 STEP 2540 H=2540:00 +CPU= 15.826
  19:26:47 STEP 2541 H=2541:00 +CPU= 23.835
  19:27:03 STEP 2542 H=2542:00 +CPU= 15.876
  19:27:19 STEP 2543 H=2543:00 +CPU= 16.000
  19:27:48 STEP 2544 H=2544:00 +CPU= 28.575
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002520
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002520
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002520
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002532
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002532
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002532
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_105.zip
Uploading the intermediate file: upload_file_105.zip
  19:28:04 STEP 2545 H=2545:00 +CPU= 15.955
  19:28:20 STEP 2546 H=2546:00 +CPU= 15.954
  19:28:44 STEP 2547 H=2547:00 +CPU= 23.684
  19:29:00 STEP 2548 H=2548:00 +CPU= 16.017
  19:29:15 STEP 2549 H=2549:00 +CPU= 15.415
  19:29:40 STEP 2550 H=2550:00 +CPU= 24.273
  19:29:56 STEP 2551 H=2551:00 +CPU= 15.435
  19:30:12 STEP 2552 H=2552:00 +CPU= 16.054
  19:30:37 STEP 2553 H=2553:00 +CPU= 24.433
  19:30:55 STEP 2554 H=2554:00 +CPU= 17.339
  19:31:12 STEP 2555 H=2555:00 +CPU= 16.546
  19:31:40 STEP 2556 H=2556:00 +CPU= 27.471
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002556
  19:31:57 STEP 2557 H=2557:00 +CPU= 16.327
  19:32:15 STEP 2558 H=2558:00 +CPU= 17.104
  19:32:40 STEP 2559 H=2559:00 +CPU= 24.860
  19:32:57 STEP 2560 H=2560:00 +CPU= 16.481
  19:33:14 STEP 2561 H=2561:00 +CPU= 16.293
  19:33:41 STEP 2562 H=2562:00 +CPU= 25.637
  19:33:56 STEP 2563 H=2563:00 +CPU= 15.515
  19:34:13 STEP 2564 H=2564:00 +CPU= 16.029
  19:34:37 STEP 2565 H=2565:00 +CPU= 24.305
  19:34:56 STEP 2566 H=2566:00 +CPU= 17.408
  19:35:14 STEP 2567 H=2567:00 +CPU= 17.437
  19:35:45 STEP 2568 H=2568:00 +CPU= 29.125
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002544
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002544
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002544
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002556
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002556
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002556
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_106.zip
Uploading the intermediate file: upload_file_106.zip
  19:36:01 STEP 2569 H=2569:00 +CPU= 15.944
  19:36:18 STEP 2570 H=2570:00 +CPU= 16.792
  19:36:43 STEP 2571 H=2571:00 +CPU= 24.395
  19:36:59 STEP 2572 H=2572:00 +CPU= 16.109
  19:37:18 STEP 2573 H=2573:00 +CPU= 18.225
  19:37:44 STEP 2574 H=2574:00 +CPU= 26.262
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.024] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4795MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.026] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.026] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.026] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.026] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.026] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4795 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193749:1675190269:283218.027] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193750:1675190270:283219.027] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1578:1578] [20230131:193750:1675190270:283219.027] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DB46A7  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000103282E  cloudsc_                 2946  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F1A790  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0553
(argv4) batchid: 973
(argv5) wuid: 12190197
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/boinc-client/slots/0
Project directory is: /home/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197
..mkdir for temp folder for results failed
Copying: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8faea2a9b983146710a477035ac73d78 to: /home/boinc-client/slots/0/oifs_43r3_ps_0553_2004050100_123_973_12190197.zip
Unzipping the namelist zip file: /home/boinc-client/slots/0/oifs_43r3_ps_0553_2004050100_123_973_12190197.zip
ic_ancil_file: ic_ancil_12190197
ifsdata_file: ifsdata_12190197
climate_data_file: clim_data_12190197
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_164efbb6da1fef1587acbeea4ac4204b to: /home/boinc-client/slots/0/ic_ancil_12190197.zip
Unzipping the IC ancils zip file: /home/boinc-client/slots/0/ic_ancil_12190197.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_d7534339accfcb785e9a4b3add0e3dbb to: /home/boinc-client/slots/0/ifsdata/ifsdata_12190197.zip
Unzipping the ifsdata_zip file: /home/boinc-client/slots/0/ifsdata/ifsdata_12190197.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/boinc-client/slots/0/159l_2/clim_data_12190197.zip
Unzipping the climate data zip file: /home/boinc-client/slots/0/159l_2/clim_data_12190197.zip
Checking for progress XML file: /home/boinc-client/slots/0/progress_file_12190197.xml
Opened progress file ok : /home/boinc-client/slots/0/progress_file_12190197.xml
-- Model is restarting --
Adjusting last_iter, 2575, to previous model restart step.
Creating progress file: /home/boinc-client/slots/0/progress_file_12190197.xml
last_cpu_time: 51235
upload_file_number: 107
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151
The child process has been launched with process id: 1969
Executing the command: /home/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1969:1969] [20230215:175913:1676480353:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:59:43 STEP 2568 H=2568:00 +CPU= 25.040
  17:59:58 STEP 2569 H=2569:00 +CPU= 14.501
  18:00:14 STEP 2570 H=2570:00 +CPU= 15.592
  18:00:38 STEP 2571 H=2571:00 +CPU= 23.915
  18:00:54 STEP 2572 H=2572:00 +CPU= 15.950
  18:01:16 STEP 2573 H=2573:00 +CPU= 18.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:02:34 STEP 2574 H=2574:00 +CPU= 24.485
  18:02:52 STEP 2575 H=2575:00 +CPU= 16.562
  18:03:11 STEP 2576 H=2576:00 +CPU= 17.312
  18:03:39 STEP 2577 H=2577:00 +CPU= 26.380
  18:03:58 STEP 2578 H=2578:00 +CPU= 17.229
  18:04:15 STEP 2579 H=2579:00 +CPU= 16.137
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:05:45 STEP 2580 H=2580:00 +CPU= 28.465
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002580
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002580
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002580
  18:06:03 STEP 2581 H=2581:00 +CPU= 16.617
  18:06:22 STEP 2582 H=2582:00 +CPU= 17.526
  18:06:51 STEP 2583 H=2583:00 +CPU= 26.176
  18:07:09 STEP 2584 H=2584:00 +CPU= 17.012
  18:07:27 STEP 2585 H=2585:00 +CPU= 16.710
  18:07:55 STEP 2586 H=2586:00 +CPU= 26.423
  18:08:11 STEP 2587 H=2587:00 +CPU= 15.526
  18:08:27 STEP 2588 H=2588:00 +CPU= 15.762
  18:08:51 STEP 2589 H=2589:00 +CPU= 23.672
  18:09:07 STEP 2590 H=2590:00 +CPU= 15.730
  21:11:46 STEP 2591 H=2591:00 +CPU= 15.817
  21:12:14 STEP 2592 H=2592:00 +CPU= 27.843
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002592
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002592
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002592
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002568
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002568
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002568
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002580
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002580
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002580
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_107.zip
Uploading the intermediate file: upload_file_107.zip
  21:12:30 STEP 2593 H=2593:00 +CPU= 15.670
  21:12:46 STEP 2594 H=2594:00 +CPU= 15.824
  21:13:11 STEP 2595 H=2595:00 +CPU= 24.038
  21:13:27 STEP 2596 H=2596:00 +CPU= 15.740
  21:13:42 STEP 2597 H=2597:00 +CPU= 15.282
  21:14:07 STEP 2598 H=2598:00 +CPU= 24.366
  21:14:23 STEP 2599 H=2599:00 +CPU= 15.328
  21:14:39 STEP 2600 H=2600:00 +CPU= 15.816
  21:15:03 STEP 2601 H=2601:00 +CPU= 23.840
  21:15:19 STEP 2602 H=2602:00 +CPU= 16.156
  21:15:36 STEP 2603 H=2603:00 +CPU= 15.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:17:23 STEP 2604 H=2604:00 +CPU= 30.129
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002604
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002604
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002604
  21:17:39 STEP 2605 H=2605:00 +CPU= 15.271
  21:17:59 STEP 2606 H=2606:00 +CPU= 17.841
  21:18:25 STEP 2607 H=2607:00 +CPU= 25.015
  21:18:42 STEP 2608 H=2608:00 +CPU= 16.759
  21:19:00 STEP 2609 H=2609:00 +CPU= 16.248
  21:19:27 STEP 2610 H=2610:00 +CPU= 25.744
  21:19:45 STEP 2611 H=2611:00 +CPU= 16.629
  21:20:03 STEP 2612 H=2612:00 +CPU= 17.157
  21:20:32 STEP 2613 H=2613:00 +CPU= 26.317
  21:20:49 STEP 2614 H=2614:00 +CPU= 16.458
  21:21:07 STEP 2615 H=2615:00 +CPU= 16.710
  21:21:39 STEP 2616 H=2616:00 +CPU= 29.953
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002616
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002616
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002616
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002592
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002592
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002592
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMGGhq0f+002604
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMSHhq0f+002604
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12190197/ICMUAhq0f+002604
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0553_2004050100_123_973_12190197_0_r1375500151_108.zip
Uploading the intermediate file: upload_file_108.zip
  21:21:56 STEP 2617 H=2617:00 +CPU= 16.405
  21:22:13 STEP 2618 H=2618:00 +CPU= 16.508
  21:22:40 STEP 2619 H=2619:00 +CPU= 25.183
  21:22:59 STEP 2620 H=2620:00 +CPU= 17.179
  21:23:16 STEP 2621 H=2621:00 +CPU= 16.663
  21:23:44 STEP 2622 H=2622:00 +CPU= 25.845
  21:24:03 STEP 2623 H=2623:00 +CPU= 17.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:24:29 STEP 2624 H=2624:00 +CPU= 14.925
  21:24:53 STEP 2625 H=2625:00 +CPU= 23.952
  21:25:09 STEP 2626 H=2626:00 +CPU= 15.737
  21:25:25 STEP 2627 H=2627:00 +CPU= 15.350
  21:25:52 STEP 2628 H=2628:00 +CPU= 26.289
Moving to projects directory: /home/boinc-client/slots/0/ICMGGhq0f+002628
Moving to projects directory: /home/boinc-client/slots/0/ICMSHhq0f+002628
Moving to projects directory: /home/boinc-client/slots/0/ICMUAhq0f+002628
  21:26:07 STEP 2629 H=2629:00 +CPU= 15.392
  21:26:23 STEP 2630 H=2630:00 +CPU= 15.753
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:45:04 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 8,726,400 48,377 0.0055
09 May 2023 16:37:32 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 6,912,000 38,129 0.0055
09 May 2023 16:37:28 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 6,048,000 33,573 0.0056
09 May 2023 16:37:22 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 5,184,000 28,996 0.0056
09 May 2023 16:37:17 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 4,320,000 24,407 0.0056
09 May 2023 16:15:48 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 2,332,800 13,713 0.0059
09 May 2023 16:15:00 1495246 22275783 oifs_43r3_ps_0553_2004050100_123_973_12190197_0 1,209,600 6,605 0.0055


©2024 cpdn.org