climateprediction.net home page
Task 22295831

Task 22295831

Name oifs_43r3_ps_0892_1999050100_123_968_12185536_1
Workunit 12185536
Created 5 Jan 2023, 16:15:42 UTC
Sent 5 Jan 2023, 16:33:17 UTC
Report deadline 4 Feb 2023, 16:33:17 UTC
Received 14 Jan 2023, 23:55:14 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537625
Run time 10 hours 47 min 23 sec
CPU time 10 hours 37 min 39 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,697.58 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,753.01 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
36/ICMSHhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_94.zip
Uploading the intermediate file: upload_file_94.zip
  12:23:28 STEP 2281 H=2281:00 +CPU=  8.952
  12:23:39 STEP 2282 H=2282:00 +CPU= 11.051
  12:23:54 STEP 2283 H=2283:00 +CPU= 14.857
  12:24:04 STEP 2284 H=2284:00 +CPU=  9.365
  12:24:13 STEP 2285 H=2285:00 +CPU=  9.082
  12:24:28 STEP 2286 H=2286:00 +CPU= 14.941
  12:24:37 STEP 2287 H=2287:00 +CPU=  9.120
  12:24:46 STEP 2288 H=2288:00 +CPU=  9.161
  12:25:01 STEP 2289 H=2289:00 +CPU= 14.561
  12:25:10 STEP 2290 H=2290:00 +CPU=  9.392
  12:25:19 STEP 2291 H=2291:00 +CPU=  9.075
  12:25:35 STEP 2292 H=2292:00 +CPU= 16.169
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002292
  12:25:44 STEP 2293 H=2293:00 +CPU=  9.015
  12:25:54 STEP 2294 H=2294:00 +CPU=  9.320
  12:26:08 STEP 2295 H=2295:00 +CPU= 14.647
  12:26:18 STEP 2296 H=2296:00 +CPU=  9.231
  12:26:27 STEP 2297 H=2297:00 +CPU=  9.016
  12:26:41 STEP 2298 H=2298:00 +CPU= 14.793
  12:26:50 STEP 2299 H=2299:00 +CPU=  9.023
  12:27:00 STEP 2300 H=2300:00 +CPU=  9.306
  12:27:14 STEP 2301 H=2301:00 +CPU= 14.556
  12:27:24 STEP 2302 H=2302:00 +CPU=  9.278
  12:27:33 STEP 2303 H=2303:00 +CPU=  9.280
  12:27:51 STEP 2304 H=2304:00 +CPU= 17.238
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_95.zip
Uploading the intermediate file: upload_file_95.zip
  12:28:00 STEP 2305 H=2305:00 +CPU=  8.941
  12:28:09 STEP 2306 H=2306:00 +CPU=  9.318
Uploading trickle at timestep: 8294400
  12:28:24 STEP 2307 H=2307:00 +CPU= 14.601
  12:28:33 STEP 2308 H=2308:00 +CPU=  9.408
  12:28:43 STEP 2309 H=2309:00 +CPU=  9.185
  12:28:58 STEP 2310 H=2310:00 +CPU= 14.988
  12:29:07 STEP 2311 H=2311:00 +CPU=  9.206
  12:29:16 STEP 2312 H=2312:00 +CPU=  9.403
  12:29:33 STEP 2313 H=2313:00 +CPU= 16.280
  12:29:42 STEP 2314 H=2314:00 +CPU=  9.533
  12:29:51 STEP 2315 H=2315:00 +CPU=  9.089
  12:30:08 STEP 2316 H=2316:00 +CPU= 16.219
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002316
  12:30:17 STEP 2317 H=2317:00 +CPU=  9.058
  12:30:26 STEP 2318 H=2318:00 +CPU=  9.420
  12:30:41 STEP 2319 H=2319:00 +CPU= 14.764
  12:30:50 STEP 2320 H=2320:00 +CPU=  9.309
  12:30:59 STEP 2321 H=2321:00 +CPU=  9.151
  12:31:14 STEP 2322 H=2322:00 +CPU= 15.202
  12:31:24 STEP 2323 H=2323:00 +CPU=  9.156
  12:31:33 STEP 2324 H=2324:00 +CPU=  9.397
  12:31:48 STEP 2325 H=2325:00 +CPU= 14.834
  12:31:57 STEP 2326 H=2326:00 +CPU=  9.492
  12:32:07 STEP 2327 H=2327:00 +CPU=  9.411
  12:32:27 STEP 2328 H=2328:00 +CPU= 18.197
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_96.zip
Uploading the intermediate file: upload_file_96.zip
  12:32:36 STEP 2329 H=2329:00 +CPU=  9.502
  12:32:46 STEP 2330 H=2330:00 +CPU=  9.477
  12:33:02 STEP 2331 H=2331:00 +CPU= 14.770
  12:33:12 STEP 2332 H=2332:00 +CPU=  9.491
  12:33:21 STEP 2333 H=2333:00 +CPU=  9.087
  12:33:36 STEP 2334 H=2334:00 +CPU= 15.063
  12:33:45 STEP 2335 H=2335:00 +CPU=  9.286
  12:33:55 STEP 2336 H=2336:00 +CPU=  9.314
  12:34:10 STEP 2337 H=2337:00 +CPU= 14.767
  12:34:19 STEP 2338 H=2338:00 +CPU=  9.448
  12:34:28 STEP 2339 H=2339:00 +CPU=  9.150
  12:34:45 STEP 2340 H=2340:00 +CPU= 16.349
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002340
  12:34:54 STEP 2341 H=2341:00 +CPU=  9.101
  12:35:03 STEP 2342 H=2342:00 +CPU=  9.413
  12:35:18 STEP 2343 H=2343:00 +CPU= 14.650
  12:35:27 STEP 2344 H=2344:00 +CPU=  9.287
  12:35:36 STEP 2345 H=2345:00 +CPU=  9.047
  12:35:51 STEP 2346 H=2346:00 +CPU= 14.955
  12:36:00 STEP 2347 H=2347:00 +CPU=  9.053
  12:36:11 STEP 2348 H=2348:00 +CPU= 10.853
  12:36:26 STEP 2349 H=2349:00 +CPU= 14.856
  12:36:35 STEP 2350 H=2350:00 +CPU=  9.263
  12:36:44 STEP 2351 H=2351:00 +CPU=  9.275
  12:37:02 STEP 2352 H=2352:00 +CPU= 17.126
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_97.zip
Uploading the intermediate file: upload_file_97.zip
  12:37:11 STEP 2353 H=2353:00 +CPU=  8.961
  12:37:20 STEP 2354 H=2354:00 +CPU=  9.227
  12:37:35 STEP 2355 H=2355:00 +CPU= 14.581
  12:37:44 STEP 2356 H=2356:00 +CPU=  9.380
  12:37:53 STEP 2357 H=2357:00 +CPU=  8.960
  12:38:08 STEP 2358 H=2358:00 +CPU= 14.826
  12:38:17 STEP 2359 H=2359:00 +CPU=  9.052
  12:38:26 STEP 2360 H=2360:00 +CPU=  9.281
  12:38:41 STEP 2361 H=2361:00 +CPU= 14.505
  12:38:50 STEP 2362 H=2362:00 +CPU=  9.365
  12:38:59 STEP 2363 H=2363:00 +CPU=  9.076
  12:39:15 STEP 2364 H=2364:00 +CPU= 16.149
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002364
  12:39:24 STEP 2365 H=2365:00 +CPU=  9.043
  12:39:36 STEP 2366 H=2366:00 +CPU= 11.268
  12:39:50 STEP 2367 H=2367:00 +CPU= 14.635
  12:39:59 STEP 2368 H=2368:00 +CPU=  9.234
  12:40:08 STEP 2369 H=2369:00 +CPU=  8.960
  12:40:23 STEP 2370 H=2370:00 +CPU= 14.855
  12:40:32 STEP 2371 H=2371:00 +CPU=  8.975
  12:40:41 STEP 2372 H=2372:00 +CPU=  9.228
  12:40:56 STEP 2373 H=2373:00 +CPU= 14.647
  12:41:05 STEP 2374 H=2374:00 +CPU=  9.230
  12:41:15 STEP 2375 H=2375:00 +CPU=  9.338
  12:41:32 STEP 2376 H=2376:00 +CPU= 17.083
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_98.zip
Uploading the intermediate file: upload_file_98.zip
  12:41:41 STEP 2377 H=2377:00 +CPU=  9.033
  12:41:50 STEP 2378 H=2378:00 +CPU=  9.255
  12:42:05 STEP 2379 H=2379:00 +CPU= 14.609
  12:42:14 STEP 2380 H=2380:00 +CPU=  9.257
  12:42:23 STEP 2381 H=2381:00 +CPU=  8.912
  12:42:38 STEP 2382 H=2382:00 +CPU= 14.927
  12:42:47 STEP 2383 H=2383:00 +CPU=  9.072
  12:42:56 STEP 2384 H=2384:00 +CPU=  9.277
  12:43:11 STEP 2385 H=2385:00 +CPU= 14.737
  12:43:20 STEP 2386 H=2386:00 +CPU=  9.359
  12:43:31 STEP 2387 H=2387:00 +CPU= 10.573
  12:43:47 STEP 2388 H=2388:00 +CPU= 16.536
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002388
  12:43:57 STEP 2389 H=2389:00 +CPU=  9.008
  12:44:06 STEP 2390 H=2390:00 +CPU=  9.260
  12:44:20 STEP 2391 H=2391:00 +CPU= 14.620
  12:44:30 STEP 2392 H=2392:00 +CPU=  9.257
  12:44:39 STEP 2393 H=2393:00 +CPU=  9.040
  12:44:54 STEP 2394 H=2394:00 +CPU= 14.870
  12:45:03 STEP 2395 H=2395:00 +CPU=  9.005
  12:45:12 STEP 2396 H=2396:00 +CPU=  9.263
  12:45:27 STEP 2397 H=2397:00 +CPU= 14.723
  12:45:36 STEP 2398 H=2398:00 +CPU=  9.385
  12:45:46 STEP 2399 H=2399:00 +CPU=  9.553
  12:46:03 STEP 2400 H=2400:00 +CPU= 17.405
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_99.zip
Uploading the intermediate file: upload_file_99.zip
  12:46:12 STEP 2401 H=2401:00 +CPU=  9.085
  12:46:22 STEP 2402 H=2402:00 +CPU=  9.846
  12:46:38 STEP 2403 H=2403:00 +CPU= 15.856
  12:46:48 STEP 2404 H=2404:00 +CPU=  9.449
  12:46:57 STEP 2405 H=2405:00 +CPU=  9.115
  12:47:12 STEP 2406 H=2406:00 +CPU= 15.097
  12:47:21 STEP 2407 H=2407:00 +CPU=  9.221
  12:47:31 STEP 2408 H=2408:00 +CPU=  9.523
  12:47:46 STEP 2409 H=2409:00 +CPU= 14.934
  12:47:55 STEP 2410 H=2410:00 +CPU=  9.488
  12:48:04 STEP 2411 H=2411:00 +CPU=  9.201
  12:48:21 STEP 2412 H=2412:00 +CPU= 16.571
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002412
  12:48:31 STEP 2413 H=2413:00 +CPU=  8.984
  12:48:40 STEP 2414 H=2414:00 +CPU=  9.341
  12:48:54 STEP 2415 H=2415:00 +CPU= 14.529
  12:49:05 STEP 2416 H=2416:00 +CPU= 10.191
  12:49:14 STEP 2417 H=2417:00 +CPU=  9.629
  12:49:29 STEP 2418 H=2418:00 +CPU= 14.896
  12:49:38 STEP 2419 H=2419:00 +CPU=  8.973
  12:49:47 STEP 2420 H=2420:00 +CPU=  9.348
  12:50:02 STEP 2421 H=2421:00 +CPU= 14.570
  12:50:11 STEP 2422 H=2422:00 +CPU=  9.208
  12:50:21 STEP 2423 H=2423:00 +CPU=  9.318
  12:50:38 STEP 2424 H=2424:00 +CPU= 17.014
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_100.zip
Uploading the intermediate file: upload_file_100.zip
  12:50:48 STEP 2425 H=2425:00 +CPU=  9.014
  12:50:57 STEP 2426 H=2426:00 +CPU=  9.319
  12:51:12 STEP 2427 H=2427:00 +CPU= 14.489
  12:51:21 STEP 2428 H=2428:00 +CPU=  9.338
  12:51:30 STEP 2429 H=2429:00 +CPU=  9.019
  12:51:45 STEP 2430 H=2430:00 +CPU= 14.951
  12:51:54 STEP 2431 H=2431:00 +CPU=  9.159
  12:52:04 STEP 2432 H=2432:00 +CPU=  9.332
  12:52:19 STEP 2433 H=2433:00 +CPU= 14.972
  12:52:30 STEP 2434 H=2434:00 +CPU= 10.858
  12:52:39 STEP 2435 H=2435:00 +CPU=  9.155
  12:52:55 STEP 2436 H=2436:00 +CPU= 16.238
  12:53:04 STEP 2437 H=2437:00 +CPU=  8.959
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002436
  12:53:13 STEP 2438 H=2438:00 +CPU=  9.304
  12:53:28 STEP 2439 H=2439:00 +CPU= 14.610
  12:53:37 STEP 2440 H=2440:00 +CPU=  9.245
  12:53:46 STEP 2441 H=2441:00 +CPU=  9.026
  12:54:01 STEP 2442 H=2442:00 +CPU= 14.943
  12:54:10 STEP 2443 H=2443:00 +CPU=  9.042
  12:54:20 STEP 2444 H=2444:00 +CPU=  9.404
  12:54:34 STEP 2445 H=2445:00 +CPU= 14.563
  12:54:43 STEP 2446 H=2446:00 +CPU=  9.236
  12:54:53 STEP 2447 H=2447:00 +CPU=  9.418
  12:55:11 STEP 2448 H=2448:00 +CPU= 17.042
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_101.zip
Uploading the intermediate file: upload_file_101.zip
  12:55:21 STEP 2449 H=2449:00 +CPU=  9.076
  12:55:30 STEP 2450 H=2450:00 +CPU=  9.262
  12:55:46 STEP 2451 H=2451:00 +CPU= 15.695
  12:55:55 STEP 2452 H=2452:00 +CPU=  9.610
  12:56:04 STEP 2453 H=2453:00 +CPU=  9.076
  12:56:19 STEP 2454 H=2454:00 +CPU= 14.818
  12:56:28 STEP 2455 H=2455:00 +CPU=  9.065
  12:56:38 STEP 2456 H=2456:00 +CPU=  9.375
  12:56:52 STEP 2457 H=2457:00 +CPU= 14.509
  12:57:01 STEP 2458 H=2458:00 +CPU=  9.294
  12:57:11 STEP 2459 H=2459:00 +CPU=  9.237
  12:57:27 STEP 2460 H=2460:00 +CPU= 16.117
  12:57:36 STEP 2461 H=2461:00 +CPU=  9.049
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002460
  12:57:46 STEP 2462 H=2462:00 +CPU=  9.415
  12:58:00 STEP 2463 H=2463:00 +CPU= 14.477
  12:58:10 STEP 2464 H=2464:00 +CPU=  9.243
  12:58:19 STEP 2465 H=2465:00 +CPU=  9.210
  12:58:34 STEP 2466 H=2466:00 +CPU= 14.812
  12:58:43 STEP 2467 H=2467:00 +CPU=  8.952
  12:58:52 STEP 2468 H=2468:00 +CPU=  9.406
  12:59:07 STEP 2469 H=2469:00 +CPU= 14.554
  12:59:17 STEP 2470 H=2470:00 +CPU=  9.960
  12:59:26 STEP 2471 H=2471:00 +CPU=  9.938
  12:59:47 STEP 2472 H=2472:00 +CPU= 17.035
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_102.zip
Uploading the intermediate file: upload_file_102.zip
  12:59:56 STEP 2473 H=2473:00 +CPU=  9.181
  13:00:06 STEP 2474 H=2474:00 +CPU=  9.199
  13:00:20 STEP 2475 H=2475:00 +CPU= 14.571
  13:00:30 STEP 2476 H=2476:00 +CPU=  9.386
  13:00:39 STEP 2477 H=2477:00 +CPU=  9.007
  13:00:53 STEP 2478 H=2478:00 +CPU= 14.725
  13:01:02 STEP 2479 H=2479:00 +CPU=  9.064
  13:01:12 STEP 2480 H=2480:00 +CPU=  9.375
  13:01:26 STEP 2481 H=2481:00 +CPU= 14.576
  13:01:36 STEP 2482 H=2482:00 +CPU=  9.301
  13:01:45 STEP 2483 H=2483:00 +CPU=  9.053
  13:02:03 STEP 2484 H=2484:00 +CPU= 18.292
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002484
  13:02:12 STEP 2485 H=2485:00 +CPU=  9.078
  13:02:22 STEP 2486 H=2486:00 +CPU=  9.387
  13:02:36 STEP 2487 H=2487:00 +CPU= 14.403
  13:02:45 STEP 2488 H=2488:00 +CPU=  9.290
  13:02:54 STEP 2489 H=2489:00 +CPU=  9.006
  13:03:09 STEP 2490 H=2490:00 +CPU= 14.754
  13:03:18 STEP 2491 H=2491:00 +CPU=  8.945
  13:03:27 STEP 2492 H=2492:00 +CPU=  9.215
  13:03:42 STEP 2493 H=2493:00 +CPU= 14.409
  13:03:51 STEP 2494 H=2494:00 +CPU=  9.150
  13:04:00 STEP 2495 H=2495:00 +CPU=  9.291
  13:04:19 STEP 2496 H=2496:00 +CPU= 17.369
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_103.zip
  13:04:30 STEP 2497 H=2497:00 +CPU= 10.441
Uploading the intermediate file: upload_file_103.zip
  13:04:39 STEP 2498 H=2498:00 +CPU=  9.220
  13:04:54 STEP 2499 H=2499:00 +CPU= 14.527
  13:05:03 STEP 2500 H=2500:00 +CPU=  9.187
  13:05:12 STEP 2501 H=2501:00 +CPU=  9.115
  13:05:27 STEP 2502 H=2502:00 +CPU= 14.870
  13:05:36 STEP 2503 H=2503:00 +CPU=  8.966
  13:05:45 STEP 2504 H=2504:00 +CPU=  9.219
  13:06:00 STEP 2505 H=2505:00 +CPU= 14.572
  13:06:09 STEP 2506 H=2506:00 +CPU=  9.234
  13:06:18 STEP 2507 H=2507:00 +CPU=  9.002
  13:06:34 STEP 2508 H=2508:00 +CPU= 16.333
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002508
  13:06:43 STEP 2509 H=2509:00 +CPU=  8.939
  13:06:52 STEP 2510 H=2510:00 +CPU=  9.185
  13:07:07 STEP 2511 H=2511:00 +CPU= 14.552
  13:07:16 STEP 2512 H=2512:00 +CPU=  9.149
  13:07:25 STEP 2513 H=2513:00 +CPU=  9.013
  13:07:40 STEP 2514 H=2514:00 +CPU= 14.767
  13:07:49 STEP 2515 H=2515:00 +CPU=  8.832
  13:07:59 STEP 2516 H=2516:00 +CPU= 10.289
  13:08:14 STEP 2517 H=2517:00 +CPU= 14.547
  13:08:23 STEP 2518 H=2518:00 +CPU=  9.186
  13:08:32 STEP 2519 H=2519:00 +CPU=  9.336
  13:08:51 STEP 2520 H=2520:00 +CPU= 16.941
  13:09:00 STEP 2521 H=2521:00 +CPU=  8.994
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_104.zip
Uploading the intermediate file: upload_file_104.zip
  13:09:09 STEP 2522 H=2522:00 +CPU=  9.140
  13:09:23 STEP 2523 H=2523:00 +CPU= 14.426
  13:09:33 STEP 2524 H=2524:00 +CPU=  9.269
  13:09:42 STEP 2525 H=2525:00 +CPU=  8.999
  13:09:57 STEP 2526 H=2526:00 +CPU= 14.761
  13:10:06 STEP 2527 H=2527:00 +CPU=  9.099
  13:10:15 STEP 2528 H=2528:00 +CPU=  9.347
  13:10:30 STEP 2529 H=2529:00 +CPU= 14.577
  13:10:39 STEP 2530 H=2530:00 +CPU=  9.337
  13:10:48 STEP 2531 H=2531:00 +CPU=  9.060
  13:11:04 STEP 2532 H=2532:00 +CPU= 16.099
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002532
  13:11:13 STEP 2533 H=2533:00 +CPU=  9.061
  13:11:24 STEP 2534 H=2534:00 +CPU= 10.729
  13:11:39 STEP 2535 H=2535:00 +CPU= 15.005
  13:11:48 STEP 2536 H=2536:00 +CPU=  9.293
  13:11:57 STEP 2537 H=2537:00 +CPU=  8.993
  13:12:12 STEP 2538 H=2538:00 +CPU= 14.824
  13:12:21 STEP 2539 H=2539:00 +CPU=  9.037
  13:12:30 STEP 2540 H=2540:00 +CPU=  9.167
  13:12:45 STEP 2541 H=2541:00 +CPU= 14.462
  13:12:54 STEP 2542 H=2542:00 +CPU=  9.209
  13:13:03 STEP 2543 H=2543:00 +CPU=  9.232
  13:13:22 STEP 2544 H=2544:00 +CPU= 17.111
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_105.zip
Uploading the intermediate file: upload_file_105.zip
  13:13:31 STEP 2545 H=2545:00 +CPU=  9.052
  13:13:41 STEP 2546 H=2546:00 +CPU=  9.342
Uploading trickle at timestep: 9158400
  13:13:55 STEP 2547 H=2547:00 +CPU= 14.640
  13:14:04 STEP 2548 H=2548:00 +CPU=  9.115
  13:14:13 STEP 2549 H=2549:00 +CPU=  9.007
  13:14:48 STEP 2550 H=2550:00 +CPU= 16.326
  13:14:57 STEP 2551 H=2551:00 +CPU=  9.098
  13:15:06 STEP 2552 H=2552:00 +CPU=  9.229
  13:15:21 STEP 2553 H=2553:00 +CPU= 14.468
  13:15:30 STEP 2554 H=2554:00 +CPU=  9.221
  13:15:39 STEP 2555 H=2555:00 +CPU=  9.064
  13:15:55 STEP 2556 H=2556:00 +CPU= 16.182
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002556
  13:16:04 STEP 2557 H=2557:00 +CPU=  9.072
  13:16:14 STEP 2558 H=2558:00 +CPU=  9.311
  13:16:28 STEP 2559 H=2559:00 +CPU= 14.613
  13:16:38 STEP 2560 H=2560:00 +CPU=  9.247
  13:16:47 STEP 2561 H=2561:00 +CPU=  8.919
  13:17:01 STEP 2562 H=2562:00 +CPU= 14.806
  13:17:10 STEP 2563 H=2563:00 +CPU=  8.955
  13:17:20 STEP 2564 H=2564:00 +CPU=  9.183
  13:17:34 STEP 2565 H=2565:00 +CPU= 14.664
  13:17:44 STEP 2566 H=2566:00 +CPU=  9.375
  13:17:55 STEP 2567 H=2567:00 +CPU= 11.006
  13:18:12 STEP 2568 H=2568:00 +CPU= 16.995
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_106.zip
Uploading the intermediate file: upload_file_106.zip
  13:18:21 STEP 2569 H=2569:00 +CPU=  8.783
  13:18:30 STEP 2570 H=2570:00 +CPU=  9.254
  13:18:45 STEP 2571 H=2571:00 +CPU= 14.131
  13:18:54 STEP 2572 H=2572:00 +CPU=  8.946
  13:19:02 STEP 2573 H=2573:00 +CPU=  8.649
  13:19:16 STEP 2574 H=2574:00 +CPU= 13.848
  13:19:25 STEP 2575 H=2575:00 +CPU=  8.475
  13:19:33 STEP 2576 H=2576:00 +CPU=  8.685
  13:19:47 STEP 2577 H=2577:00 +CPU= 13.405
  13:19:55 STEP 2578 H=2578:00 +CPU=  8.620
  13:20:04 STEP 2579 H=2579:00 +CPU=  8.364
  13:20:19 STEP 2580 H=2580:00 +CPU= 15.076
  13:20:27 STEP 2581 H=2581:00 +CPU=  8.492
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002580
  13:20:36 STEP 2582 H=2582:00 +CPU=  8.749
  13:20:49 STEP 2583 H=2583:00 +CPU= 13.433
  13:20:58 STEP 2584 H=2584:00 +CPU=  8.630
  13:21:07 STEP 2585 H=2585:00 +CPU=  8.531
  13:21:20 STEP 2586 H=2586:00 +CPU= 13.678
  13:21:29 STEP 2587 H=2587:00 +CPU=  8.411
  13:21:37 STEP 2588 H=2588:00 +CPU=  8.691
  13:21:51 STEP 2589 H=2589:00 +CPU= 13.333
  13:21:59 STEP 2590 H=2590:00 +CPU=  8.571
  13:22:08 STEP 2591 H=2591:00 +CPU=  8.714
  13:22:24 STEP 2592 H=2592:00 +CPU= 15.740
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_107.zip
Uploading the intermediate file: upload_file_107.zip
  13:22:33 STEP 2593 H=2593:00 +CPU=  8.436
  13:22:41 STEP 2594 H=2594:00 +CPU=  8.657
  13:22:55 STEP 2595 H=2595:00 +CPU= 13.559
  13:23:03 STEP 2596 H=2596:00 +CPU=  8.681
  13:23:12 STEP 2597 H=2597:00 +CPU=  8.449
  13:23:28 STEP 2598 H=2598:00 +CPU= 15.653
  13:23:36 STEP 2599 H=2599:00 +CPU=  8.830
  13:23:45 STEP 2600 H=2600:00 +CPU=  8.924
  13:23:59 STEP 2601 H=2601:00 +CPU= 13.896
  13:24:08 STEP 2602 H=2602:00 +CPU=  9.069
  13:24:17 STEP 2603 H=2603:00 +CPU=  8.876
  13:24:33 STEP 2604 H=2604:00 +CPU= 15.504
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002604
  13:24:41 STEP 2605 H=2605:00 +CPU=  8.798
  13:24:50 STEP 2606 H=2606:00 +CPU=  8.955
  13:25:04 STEP 2607 H=2607:00 +CPU= 13.912
  13:25:13 STEP 2608 H=2608:00 +CPU=  9.006
  13:25:22 STEP 2609 H=2609:00 +CPU=  8.880
  13:25:36 STEP 2610 H=2610:00 +CPU= 14.097
  13:25:45 STEP 2611 H=2611:00 +CPU=  8.678
  13:25:54 STEP 2612 H=2612:00 +CPU=  8.868
  13:26:08 STEP 2613 H=2613:00 +CPU= 13.861
  13:26:17 STEP 2614 H=2614:00 +CPU=  8.950
  13:26:26 STEP 2615 H=2615:00 +CPU=  8.913
  13:26:42 STEP 2616 H=2616:00 +CPU= 16.461
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_108.zip
Uploading the intermediate file: upload_file_108.zip
  13:26:51 STEP 2617 H=2617:00 +CPU=  8.730
  13:27:00 STEP 2618 H=2618:00 +CPU=  8.992
  13:27:14 STEP 2619 H=2619:00 +CPU= 13.891
  13:27:23 STEP 2620 H=2620:00 +CPU=  8.886
  13:27:31 STEP 2621 H=2621:00 +CPU=  8.724
  13:27:46 STEP 2622 H=2622:00 +CPU= 14.127
  13:27:54 STEP 2623 H=2623:00 +CPU=  8.805
  13:28:03 STEP 2624 H=2624:00 +CPU=  9.038
  13:28:17 STEP 2625 H=2625:00 +CPU= 13.850
  13:28:26 STEP 2626 H=2626:00 +CPU=  8.955
  13:28:35 STEP 2627 H=2627:00 +CPU=  8.815
  13:28:53 STEP 2628 H=2628:00 +CPU= 17.447
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002628
  13:29:01 STEP 2629 H=2629:00 +CPU=  8.694
  13:29:10 STEP 2630 H=2630:00 +CPU=  9.026
  13:29:24 STEP 2631 H=2631:00 +CPU= 13.926
  13:29:33 STEP 2632 H=2632:00 +CPU=  8.962
  13:29:42 STEP 2633 H=2633:00 +CPU=  8.760
  13:29:56 STEP 2634 H=2634:00 +CPU= 14.162
  13:30:05 STEP 2635 H=2635:00 +CPU=  8.675
  13:30:14 STEP 2636 H=2636:00 +CPU=  8.947
  13:30:28 STEP 2637 H=2637:00 +CPU= 13.936
  13:30:37 STEP 2638 H=2638:00 +CPU=  8.996
  13:30:46 STEP 2639 H=2639:00 +CPU=  8.971
  13:31:02 STEP 2640 H=2640:00 +CPU= 16.446
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_109.zip
Uploading the intermediate file: upload_file_109.zip
  13:31:11 STEP 2641 H=2641:00 +CPU=  8.844
  13:31:20 STEP 2642 H=2642:00 +CPU=  8.920
  13:31:34 STEP 2643 H=2643:00 +CPU= 13.868
  13:31:43 STEP 2644 H=2644:00 +CPU=  8.976
  13:31:51 STEP 2645 H=2645:00 +CPU=  8.703
  13:32:07 STEP 2646 H=2646:00 +CPU= 15.199
  13:32:16 STEP 2647 H=2647:00 +CPU=  8.883
  13:32:25 STEP 2648 H=2648:00 +CPU=  8.998
  13:32:39 STEP 2649 H=2649:00 +CPU= 14.023
  13:32:48 STEP 2650 H=2650:00 +CPU=  9.057
  13:32:57 STEP 2651 H=2651:00 +CPU=  8.863
  13:33:12 STEP 2652 H=2652:00 +CPU= 15.511
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002652
  13:33:21 STEP 2653 H=2653:00 +CPU=  8.656
  13:33:30 STEP 2654 H=2654:00 +CPU=  8.959
  13:33:44 STEP 2655 H=2655:00 +CPU= 13.904
  13:33:53 STEP 2656 H=2656:00 +CPU=  8.949
  13:34:01 STEP 2657 H=2657:00 +CPU=  8.724
  13:34:16 STEP 2658 H=2658:00 +CPU= 14.267
  13:34:24 STEP 2659 H=2659:00 +CPU=  8.847
  13:34:33 STEP 2660 H=2660:00 +CPU=  8.981
  13:34:47 STEP 2661 H=2661:00 +CPU= 13.993
  13:34:56 STEP 2662 H=2662:00 +CPU=  9.013
  13:35:05 STEP 2663 H=2663:00 +CPU=  9.012
  13:35:22 STEP 2664 H=2664:00 +CPU= 16.400
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_110.zip
Uploading the intermediate file: upload_file_110.zip
  13:35:30 STEP 2665 H=2665:00 +CPU=  8.631
  13:35:39 STEP 2666 H=2666:00 +CPU=  8.895
  13:35:55 STEP 2667 H=2667:00 +CPU= 15.803
  13:36:04 STEP 2668 H=2668:00 +CPU=  8.924
  13:36:13 STEP 2669 H=2669:00 +CPU=  8.628
  13:36:27 STEP 2670 H=2670:00 +CPU= 14.244
  13:36:36 STEP 2671 H=2671:00 +CPU=  8.647
  13:36:45 STEP 2672 H=2672:00 +CPU=  9.032
  13:36:59 STEP 2673 H=2673:00 +CPU= 14.108
  13:37:08 STEP 2674 H=2674:00 +CPU=  8.946
  13:37:16 STEP 2675 H=2675:00 +CPU=  8.738
  13:37:32 STEP 2676 H=2676:00 +CPU= 15.590
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002676
  13:37:41 STEP 2677 H=2677:00 +CPU=  8.768
  13:37:50 STEP 2678 H=2678:00 +CPU=  8.988
  13:38:04 STEP 2679 H=2679:00 +CPU= 13.856
  13:38:13 STEP 2680 H=2680:00 +CPU=  8.976
  13:38:21 STEP 2681 H=2681:00 +CPU=  8.693
  13:38:35 STEP 2682 H=2682:00 +CPU= 14.115
  13:38:44 STEP 2683 H=2683:00 +CPU=  8.686
  13:38:53 STEP 2684 H=2684:00 +CPU=  8.930
  13:39:09 STEP 2685 H=2685:00 +CPU= 15.536
  13:39:18 STEP 2686 H=2686:00 +CPU=  8.949
  13:39:27 STEP 2687 H=2687:00 +CPU=  8.983
  13:39:43 STEP 2688 H=2688:00 +CPU= 16.358
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_111.zip
Uploading the intermediate file: upload_file_111.zip
  13:39:52 STEP 2689 H=2689:00 +CPU=  8.677
  13:40:01 STEP 2690 H=2690:00 +CPU=  9.028
  13:40:15 STEP 2691 H=2691:00 +CPU= 13.979
  13:40:24 STEP 2692 H=2692:00 +CPU=  9.020
  13:40:32 STEP 2693 H=2693:00 +CPU=  8.687
  13:40:47 STEP 2694 H=2694:00 +CPU= 14.373
  13:40:56 STEP 2695 H=2695:00 +CPU=  8.742
  13:41:05 STEP 2696 H=2696:00 +CPU=  9.043
  13:41:19 STEP 2697 H=2697:00 +CPU= 14.023
  13:41:28 STEP 2698 H=2698:00 +CPU=  9.069
  13:41:37 STEP 2699 H=2699:00 +CPU=  8.758
  13:41:53 STEP 2700 H=2700:00 +CPU= 16.577
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002700
  13:42:02 STEP 2701 H=2701:00 +CPU=  9.008
  13:42:11 STEP 2702 H=2702:00 +CPU=  8.958
  13:42:25 STEP 2703 H=2703:00 +CPU= 14.044
  13:42:34 STEP 2704 H=2704:00 +CPU=  9.013
  13:42:43 STEP 2705 H=2705:00 +CPU=  8.695
  13:42:57 STEP 2706 H=2706:00 +CPU= 14.238
  13:43:06 STEP 2707 H=2707:00 +CPU=  8.729
  13:43:15 STEP 2708 H=2708:00 +CPU=  8.863
  13:43:29 STEP 2709 H=2709:00 +CPU= 14.066
  13:43:38 STEP 2710 H=2710:00 +CPU=  8.932
  13:43:47 STEP 2711 H=2711:00 +CPU=  8.951
  13:44:03 STEP 2712 H=2712:00 +CPU= 16.355
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_112.zip
Uploading the intermediate file: upload_file_112.zip
  13:44:12 STEP 2713 H=2713:00 +CPU=  8.648
  13:44:21 STEP 2714 H=2714:00 +CPU=  8.928
  13:44:35 STEP 2715 H=2715:00 +CPU= 13.892
  13:44:44 STEP 2716 H=2716:00 +CPU=  8.992
  13:44:52 STEP 2717 H=2717:00 +CPU=  8.688
  13:45:08 STEP 2718 H=2718:00 +CPU= 15.261
  13:45:17 STEP 2719 H=2719:00 +CPU=  9.060
  13:45:26 STEP 2720 H=2720:00 +CPU=  8.917
  13:45:40 STEP 2721 H=2721:00 +CPU= 14.090
  13:45:49 STEP 2722 H=2722:00 +CPU=  8.896
  13:45:58 STEP 2723 H=2723:00 +CPU=  8.805
  13:46:13 STEP 2724 H=2724:00 +CPU= 15.687
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002724
  13:46:22 STEP 2725 H=2725:00 +CPU=  8.722
  13:46:31 STEP 2726 H=2726:00 +CPU=  9.005
  13:46:45 STEP 2727 H=2727:00 +CPU= 14.025
  13:46:54 STEP 2728 H=2728:00 +CPU=  8.998
  13:47:03 STEP 2729 H=2729:00 +CPU=  8.771
  13:47:17 STEP 2730 H=2730:00 +CPU= 14.292
  13:47:26 STEP 2731 H=2731:00 +CPU=  8.827
  13:47:35 STEP 2732 H=2732:00 +CPU=  9.174
  13:47:49 STEP 2733 H=2733:00 +CPU= 14.200
  13:47:58 STEP 2734 H=2734:00 +CPU=  9.031
  13:48:08 STEP 2735 H=2735:00 +CPU=  9.231
  13:48:25 STEP 2736 H=2736:00 +CPU= 17.491
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_113.zip
Uploading the intermediate file: upload_file_113.zip
  13:48:34 STEP 2737 H=2737:00 +CPU=  8.687
  13:48:43 STEP 2738 H=2738:00 +CPU=  9.128
  13:48:57 STEP 2739 H=2739:00 +CPU= 14.062
  13:49:06 STEP 2740 H=2740:00 +CPU=  9.063
  13:49:15 STEP 2741 H=2741:00 +CPU=  8.736
  13:49:29 STEP 2742 H=2742:00 +CPU= 14.618
  13:49:38 STEP 2743 H=2743:00 +CPU=  8.948
  13:49:48 STEP 2744 H=2744:00 +CPU=  9.139
  13:50:02 STEP 2745 H=2745:00 +CPU= 14.357
  13:50:11 STEP 2746 H=2746:00 +CPU=  9.043
  13:50:20 STEP 2747 H=2747:00 +CPU=  8.772
  13:50:36 STEP 2748 H=2748:00 +CPU= 15.812
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002748
  13:50:44 STEP 2749 H=2749:00 +CPU=  8.929
  13:50:54 STEP 2750 H=2750:00 +CPU=  9.141
  13:51:08 STEP 2751 H=2751:00 +CPU= 14.330
  13:51:17 STEP 2752 H=2752:00 +CPU=  9.169
  13:51:26 STEP 2753 H=2753:00 +CPU=  8.892
  13:51:42 STEP 2754 H=2754:00 +CPU= 16.131
  13:51:51 STEP 2755 H=2755:00 +CPU=  8.811
  13:52:00 STEP 2756 H=2756:00 +CPU=  9.098
  13:52:14 STEP 2757 H=2757:00 +CPU= 14.245
  13:52:23 STEP 2758 H=2758:00 +CPU=  8.981
  13:52:32 STEP 2759 H=2759:00 +CPU=  9.123
  13:52:49 STEP 2760 H=2760:00 +CPU= 16.687
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_114.zip
Uploading the intermediate file: upload_file_114.zip
  13:52:58 STEP 2761 H=2761:00 +CPU=  8.765
  13:53:07 STEP 2762 H=2762:00 +CPU=  9.008
  13:53:21 STEP 2763 H=2763:00 +CPU= 14.175
  13:53:30 STEP 2764 H=2764:00 +CPU=  9.172
  13:53:39 STEP 2765 H=2765:00 +CPU=  8.893
  13:53:54 STEP 2766 H=2766:00 +CPU= 14.602
  13:54:03 STEP 2767 H=2767:00 +CPU=  8.789
  13:54:12 STEP 2768 H=2768:00 +CPU=  9.056
  13:54:26 STEP 2769 H=2769:00 +CPU= 14.194
  13:54:35 STEP 2770 H=2770:00 +CPU=  8.977
  13:54:44 STEP 2771 H=2771:00 +CPU=  8.848
  13:55:00 STEP 2772 H=2772:00 +CPU= 16.449
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002772
  13:55:09 STEP 2773 H=2773:00 +CPU=  8.957
  13:55:18 STEP 2774 H=2774:00 +CPU=  8.965
  13:55:32 STEP 2775 H=2775:00 +CPU= 13.740
  13:55:41 STEP 2776 H=2776:00 +CPU=  8.794
  13:55:49 STEP 2777 H=2777:00 +CPU=  8.564
  13:56:03 STEP 2778 H=2778:00 +CPU= 14.165
  13:56:12 STEP 2779 H=2779:00 +CPU=  8.557
  13:56:21 STEP 2780 H=2780:00 +CPU=  8.758
  13:56:34 STEP 2781 H=2781:00 +CPU= 13.785
  13:56:43 STEP 2782 H=2782:00 +CPU=  8.897
  13:56:52 STEP 2783 H=2783:00 +CPU=  8.694
  13:57:08 STEP 2784 H=2784:00 +CPU= 16.116
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_115.zip
Uploading the intermediate file: upload_file_115.zip
  13:57:17 STEP 2785 H=2785:00 +CPU=  8.475
  13:57:25 STEP 2786 H=2786:00 +CPU=  8.711
Uploading trickle at timestep: 10022400
  13:57:39 STEP 2787 H=2787:00 +CPU= 13.630
  13:57:48 STEP 2788 H=2788:00 +CPU=  8.734
  13:57:56 STEP 2789 H=2789:00 +CPU=  8.462
  13:58:10 STEP 2790 H=2790:00 +CPU= 13.912
  13:58:18 STEP 2791 H=2791:00 +CPU=  8.470
  13:58:27 STEP 2792 H=2792:00 +CPU=  8.699
  13:58:41 STEP 2793 H=2793:00 +CPU= 13.726
  13:58:50 STEP 2794 H=2794:00 +CPU=  8.615
  13:58:58 STEP 2795 H=2795:00 +CPU=  8.375
  13:59:13 STEP 2796 H=2796:00 +CPU= 15.268
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002796
  13:59:22 STEP 2797 H=2797:00 +CPU=  8.398
  13:59:30 STEP 2798 H=2798:00 +CPU=  8.669
  13:59:44 STEP 2799 H=2799:00 +CPU= 13.650
  13:59:53 STEP 2800 H=2800:00 +CPU=  8.665
  14:00:01 STEP 2801 H=2801:00 +CPU=  8.404
  14:00:15 STEP 2802 H=2802:00 +CPU= 13.887
  14:00:23 STEP 2803 H=2803:00 +CPU=  8.359
  14:00:32 STEP 2804 H=2804:00 +CPU=  8.664
  14:00:45 STEP 2805 H=2805:00 +CPU= 13.567
  14:00:54 STEP 2806 H=2806:00 +CPU=  8.610
  14:01:03 STEP 2807 H=2807:00 +CPU=  8.547
  14:01:18 STEP 2808 H=2808:00 +CPU= 15.842
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_116.zip
Uploading the intermediate file: upload_file_116.zip
  14:01:27 STEP 2809 H=2809:00 +CPU=  8.403
  14:01:36 STEP 2810 H=2810:00 +CPU=  8.639
  14:01:49 STEP 2811 H=2811:00 +CPU= 13.494
  14:01:58 STEP 2812 H=2812:00 +CPU=  8.586
  14:02:06 STEP 2813 H=2813:00 +CPU=  8.398
  14:02:20 STEP 2814 H=2814:00 +CPU= 13.781
  14:02:28 STEP 2815 H=2815:00 +CPU=  8.348
  14:02:37 STEP 2816 H=2816:00 +CPU=  8.702
  14:02:50 STEP 2817 H=2817:00 +CPU= 13.636
  14:03:01 STEP 2818 H=2818:00 +CPU= 10.344
  14:03:09 STEP 2819 H=2819:00 +CPU=  8.658
  14:03:25 STEP 2820 H=2820:00 +CPU= 15.183
  14:03:33 STEP 2821 H=2821:00 +CPU=  8.597
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002820
  14:03:42 STEP 2822 H=2822:00 +CPU=  8.796
  14:03:56 STEP 2823 H=2823:00 +CPU= 13.673
  14:04:05 STEP 2824 H=2824:00 +CPU=  8.794
  14:04:13 STEP 2825 H=2825:00 +CPU=  8.542
  14:04:27 STEP 2826 H=2826:00 +CPU= 13.874
  14:04:35 STEP 2827 H=2827:00 +CPU=  8.468
  14:04:44 STEP 2828 H=2828:00 +CPU=  8.661
  14:04:58 STEP 2829 H=2829:00 +CPU= 13.552
  14:05:06 STEP 2830 H=2830:00 +CPU=  8.695
  14:05:15 STEP 2831 H=2831:00 +CPU=  8.666
  14:05:31 STEP 2832 H=2832:00 +CPU= 16.103
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_117.zip
Uploading the intermediate file: upload_file_117.zip
  14:05:39 STEP 2833 H=2833:00 +CPU=  8.328
  14:05:48 STEP 2834 H=2834:00 +CPU=  8.604
  14:06:02 STEP 2835 H=2835:00 +CPU= 13.511
  14:06:10 STEP 2836 H=2836:00 +CPU=  8.572
  14:06:19 STEP 2837 H=2837:00 +CPU=  8.384
  14:06:32 STEP 2838 H=2838:00 +CPU= 13.946
  14:06:41 STEP 2839 H=2839:00 +CPU=  8.290
  14:06:49 STEP 2840 H=2840:00 +CPU=  8.630
  14:07:03 STEP 2841 H=2841:00 +CPU= 13.602
  14:07:12 STEP 2842 H=2842:00 +CPU=  8.538
  14:07:20 STEP 2843 H=2843:00 +CPU=  8.451
  14:07:35 STEP 2844 H=2844:00 +CPU= 15.169
  14:07:44 STEP 2845 H=2845:00 +CPU=  8.389
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002844
  14:07:52 STEP 2846 H=2846:00 +CPU=  8.770
  14:08:06 STEP 2847 H=2847:00 +CPU= 13.521
  14:08:14 STEP 2848 H=2848:00 +CPU=  8.634
  14:08:23 STEP 2849 H=2849:00 +CPU=  8.406
  14:08:37 STEP 2850 H=2850:00 +CPU= 13.839
  14:08:45 STEP 2851 H=2851:00 +CPU=  8.332
  14:08:54 STEP 2852 H=2852:00 +CPU=  8.595
  14:09:07 STEP 2853 H=2853:00 +CPU= 13.581
  14:09:16 STEP 2854 H=2854:00 +CPU=  8.560
  14:09:24 STEP 2855 H=2855:00 +CPU=  8.621
  14:09:40 STEP 2856 H=2856:00 +CPU= 16.076
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_118.zip
Uploading the intermediate file: upload_file_118.zip
  14:09:49 STEP 2857 H=2857:00 +CPU=  8.488
  14:09:58 STEP 2858 H=2858:00 +CPU=  8.686
  14:10:11 STEP 2859 H=2859:00 +CPU= 13.562
  14:10:20 STEP 2860 H=2860:00 +CPU=  8.721
  14:10:28 STEP 2861 H=2861:00 +CPU=  8.449
  14:10:42 STEP 2862 H=2862:00 +CPU= 13.826
  14:10:51 STEP 2863 H=2863:00 +CPU=  8.357
  14:10:59 STEP 2864 H=2864:00 +CPU=  8.550
  14:11:13 STEP 2865 H=2865:00 +CPU= 13.524
  14:11:21 STEP 2866 H=2866:00 +CPU=  8.741
  14:11:30 STEP 2867 H=2867:00 +CPU=  8.364
  14:11:45 STEP 2868 H=2868:00 +CPU= 15.163
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002868
  14:11:53 STEP 2869 H=2869:00 +CPU=  8.454
  14:12:02 STEP 2870 H=2870:00 +CPU=  8.714
  14:12:16 STEP 2871 H=2871:00 +CPU= 13.489
  14:12:24 STEP 2872 H=2872:00 +CPU=  8.664
  14:12:33 STEP 2873 H=2873:00 +CPU=  8.464
  14:12:47 STEP 2874 H=2874:00 +CPU= 13.799
  14:12:55 STEP 2875 H=2875:00 +CPU=  8.371
  14:13:04 STEP 2876 H=2876:00 +CPU=  8.584
  14:13:17 STEP 2877 H=2877:00 +CPU= 13.503
  14:13:26 STEP 2878 H=2878:00 +CPU=  8.577
  14:13:34 STEP 2879 H=2879:00 +CPU=  8.581
  14:13:50 STEP 2880 H=2880:00 +CPU= 15.836
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_119.zip
Uploading the intermediate file: upload_file_119.zip
  14:13:58 STEP 2881 H=2881:00 +CPU=  8.395
  14:14:08 STEP 2882 H=2882:00 +CPU=  9.665
  14:14:22 STEP 2883 H=2883:00 +CPU= 14.062
  14:14:31 STEP 2884 H=2884:00 +CPU=  8.710
  14:14:39 STEP 2885 H=2885:00 +CPU=  8.498
  14:14:53 STEP 2886 H=2886:00 +CPU= 13.872
  14:15:02 STEP 2887 H=2887:00 +CPU=  8.310
  14:15:10 STEP 2888 H=2888:00 +CPU=  8.574
  14:15:24 STEP 2889 H=2889:00 +CPU= 13.497
  14:15:32 STEP 2890 H=2890:00 +CPU=  8.609
  14:15:41 STEP 2891 H=2891:00 +CPU=  8.352
  14:15:56 STEP 2892 H=2892:00 +CPU= 15.217
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002892
  14:16:04 STEP 2893 H=2893:00 +CPU=  8.370
  14:16:13 STEP 2894 H=2894:00 +CPU=  8.626
  14:16:26 STEP 2895 H=2895:00 +CPU= 13.549
  14:16:35 STEP 2896 H=2896:00 +CPU=  8.640
  14:16:43 STEP 2897 H=2897:00 +CPU=  8.356
  14:16:57 STEP 2898 H=2898:00 +CPU= 13.771
  14:17:06 STEP 2899 H=2899:00 +CPU=  8.319
  14:17:14 STEP 2900 H=2900:00 +CPU=  8.604
  14:17:28 STEP 2901 H=2901:00 +CPU= 13.532
  14:17:36 STEP 2902 H=2902:00 +CPU=  8.562
  14:17:45 STEP 2903 H=2903:00 +CPU=  8.574
  14:18:01 STEP 2904 H=2904:00 +CPU= 15.939
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_120.zip
Uploading the intermediate file: upload_file_120.zip
  14:18:09 STEP 2905 H=2905:00 +CPU=  8.380
  14:18:18 STEP 2906 H=2906:00 +CPU=  8.630
  14:18:31 STEP 2907 H=2907:00 +CPU= 13.584
  14:18:40 STEP 2908 H=2908:00 +CPU=  8.626
  14:18:48 STEP 2909 H=2909:00 +CPU=  8.388
  14:19:02 STEP 2910 H=2910:00 +CPU= 13.794
  14:19:11 STEP 2911 H=2911:00 +CPU=  8.326
  14:19:19 STEP 2912 H=2912:00 +CPU=  8.573
  14:19:33 STEP 2913 H=2913:00 +CPU= 13.508
  14:19:41 STEP 2914 H=2914:00 +CPU=  8.629
  14:19:50 STEP 2915 H=2915:00 +CPU=  8.335
  14:20:05 STEP 2916 H=2916:00 +CPU= 15.140
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002916
  14:20:13 STEP 2917 H=2917:00 +CPU=  8.461
  14:20:22 STEP 2918 H=2918:00 +CPU=  8.696
  14:20:35 STEP 2919 H=2919:00 +CPU= 13.506
  14:20:44 STEP 2920 H=2920:00 +CPU=  8.697
  14:20:53 STEP 2921 H=2921:00 +CPU=  8.462
  14:21:06 STEP 2922 H=2922:00 +CPU= 13.823
  14:21:15 STEP 2923 H=2923:00 +CPU=  8.330
  14:21:23 STEP 2924 H=2924:00 +CPU=  8.639
  14:21:37 STEP 2925 H=2925:00 +CPU= 13.512
  14:21:45 STEP 2926 H=2926:00 +CPU=  8.538
  14:21:54 STEP 2927 H=2927:00 +CPU=  8.594
  14:22:10 STEP 2928 H=2928:00 +CPU= 15.866
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_121.zip
Uploading the intermediate file: upload_file_121.zip
  14:22:18 STEP 2929 H=2929:00 +CPU=  8.368
  14:22:28 STEP 2930 H=2930:00 +CPU=  9.791
  14:22:42 STEP 2931 H=2931:00 +CPU= 13.995
  14:22:51 STEP 2932 H=2932:00 +CPU=  8.597
  14:22:59 STEP 2933 H=2933:00 +CPU=  8.473
  14:23:13 STEP 2934 H=2934:00 +CPU= 13.889
  14:23:21 STEP 2935 H=2935:00 +CPU=  8.378
  14:23:30 STEP 2936 H=2936:00 +CPU=  8.632
  14:23:44 STEP 2937 H=2937:00 +CPU= 13.634
  14:23:52 STEP 2938 H=2938:00 +CPU=  8.705
  14:24:01 STEP 2939 H=2939:00 +CPU=  8.587
  14:24:17 STEP 2940 H=2940:00 +CPU= 15.736
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002940
  14:24:26 STEP 2941 H=2941:00 +CPU=  9.016
  14:24:35 STEP 2942 H=2942:00 +CPU=  9.181
  14:24:49 STEP 2943 H=2943:00 +CPU= 14.102
  14:24:58 STEP 2944 H=2944:00 +CPU=  9.096
  14:25:07 STEP 2945 H=2945:00 +CPU=  8.815
  14:25:21 STEP 2946 H=2946:00 +CPU= 14.549
  14:25:30 STEP 2947 H=2947:00 +CPU=  8.757
  14:25:39 STEP 2948 H=2948:00 +CPU=  9.024
  14:25:53 STEP 2949 H=2949:00 +CPU= 14.027
  14:26:02 STEP 2950 H=2950:00 +CPU=  9.034
  14:26:11 STEP 2951 H=2951:00 +CPU=  9.045
  14:26:29 STEP 2952 H=2952:00 +CPU= 17.157
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12185536/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0892_1999050100_123_968_12185536_1_r718578732_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
14:29:52 (99): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:56 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 10,623,600 38,259 0.0036
09 May 2023 16:09:52 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 10,022,400 36,503 0.0036
09 May 2023 16:09:49 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 9,158,400 33,894 0.0037
09 May 2023 16:09:49 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 8,294,400 31,184 0.0038
09 May 2023 16:09:45 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 7,430,400 28,471 0.0038
09 May 2023 16:09:42 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 6,566,400 25,745 0.0039
09 May 2023 16:09:42 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 6,566,400 25,745 0.0039
09 May 2023 16:09:38 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 5,702,400 23,016 0.0040
09 May 2023 16:09:34 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 4,320,000 18,511 0.0043
09 May 2023 16:09:34 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 4,320,000 18,511 0.0043
09 May 2023 16:09:32 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 3,456,000 15,833 0.0046
09 May 2023 16:09:28 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 2,592,000 13,131 0.0051
09 May 2023 16:09:24 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 1,728,000 10,435 0.0060
09 May 2023 16:09:21 1537625 22295831 oifs_43r3_ps_0892_1999050100_123_968_12185536_1 864,000 5,596 0.0065


©2024 climateprediction.net