climateprediction.net home page
Task 22276464

Task 22276464

Name oifs_43r3_ps_0167_2005050100_123_974_12190811_0
Workunit 12190811
Created 23 Dec 2022, 9:52:03 UTC
Sent 5 Jan 2023, 0:08:09 UTC
Report deadline 4 Feb 2023, 0:08:09 UTC
Received 11 Jan 2023, 21:57:00 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537625
Run time 12 hours 56 min 40 sec
CPU time 12 hours 42 min 27 sec
Validate state Valid
Credit 14,394.24
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,729.46 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,752.03 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
 the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_94.zip
Uploading the intermediate file: upload_file_94.zip
  21:31:31 STEP 2281 H=2281:00 +CPU= 17.231
  21:31:50 STEP 2282 H=2282:00 +CPU= 18.993
  21:32:18 STEP 2283 H=2283:00 +CPU= 27.705
  21:32:37 STEP 2284 H=2284:00 +CPU= 18.626
  21:32:55 STEP 2285 H=2285:00 +CPU= 18.133
  21:33:21 STEP 2286 H=2286:00 +CPU= 26.415
  21:33:38 STEP 2287 H=2287:00 +CPU= 16.443
  21:33:55 STEP 2288 H=2288:00 +CPU= 17.322
  21:34:22 STEP 2289 H=2289:00 +CPU= 26.861
  21:34:40 STEP 2290 H=2290:00 +CPU= 17.753
  21:34:58 STEP 2291 H=2291:00 +CPU= 18.735
  21:35:29 STEP 2292 H=2292:00 +CPU= 30.670
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002292
  21:35:47 STEP 2293 H=2293:00 +CPU= 17.597
  21:36:07 STEP 2294 H=2294:00 +CPU= 19.992
  21:36:34 STEP 2295 H=2295:00 +CPU= 27.665
  21:36:52 STEP 2296 H=2296:00 +CPU= 18.122
  21:37:10 STEP 2297 H=2297:00 +CPU= 17.133
  21:37:38 STEP 2298 H=2298:00 +CPU= 28.205
  21:37:56 STEP 2299 H=2299:00 +CPU= 17.728
  21:38:14 STEP 2300 H=2300:00 +CPU= 18.567
  21:38:42 STEP 2301 H=2301:00 +CPU= 28.169
  21:39:00 STEP 2302 H=2302:00 +CPU= 17.467
  21:39:18 STEP 2303 H=2303:00 +CPU= 18.557
  21:39:55 STEP 2304 H=2304:00 +CPU= 30.805
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_95.zip
Uploading the intermediate file: upload_file_95.zip
  21:40:11 STEP 2305 H=2305:00 +CPU= 16.134
  21:40:28 STEP 2306 H=2306:00 +CPU= 17.070
  21:40:54 STEP 2307 H=2307:00 +CPU= 26.133
  21:41:11 STEP 2308 H=2308:00 +CPU= 17.130
  21:41:28 STEP 2309 H=2309:00 +CPU= 16.557
  21:41:55 STEP 2310 H=2310:00 +CPU= 26.969
  21:42:12 STEP 2311 H=2311:00 +CPU= 16.993
  21:42:29 STEP 2312 H=2312:00 +CPU= 17.558
  21:42:56 STEP 2313 H=2313:00 +CPU= 26.220
  21:43:13 STEP 2314 H=2314:00 +CPU= 17.470
  21:43:30 STEP 2315 H=2315:00 +CPU= 17.039
  21:43:59 STEP 2316 H=2316:00 +CPU= 29.199
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002316
  21:44:16 STEP 2317 H=2317:00 +CPU= 17.004
  21:44:33 STEP 2318 H=2318:00 +CPU= 17.153
  21:44:59 STEP 2319 H=2319:00 +CPU= 26.051
  21:45:17 STEP 2320 H=2320:00 +CPU= 17.282
  21:45:33 STEP 2321 H=2321:00 +CPU= 16.413
  21:46:00 STEP 2322 H=2322:00 +CPU= 26.623
  21:46:16 STEP 2323 H=2323:00 +CPU= 16.506
  21:46:33 STEP 2324 H=2324:00 +CPU= 16.850
  21:47:00 STEP 2325 H=2325:00 +CPU= 26.438
  21:47:17 STEP 2326 H=2326:00 +CPU= 17.151
  21:47:34 STEP 2327 H=2327:00 +CPU= 16.979
  21:48:07 STEP 2328 H=2328:00 +CPU= 31.241
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_96.zip
Uploading the intermediate file: upload_file_96.zip
  21:48:23 STEP 2329 H=2329:00 +CPU= 15.983
  21:48:41 STEP 2330 H=2330:00 +CPU= 17.081
  21:49:07 STEP 2331 H=2331:00 +CPU= 26.487
  21:49:25 STEP 2332 H=2332:00 +CPU= 17.635
  21:49:44 STEP 2333 H=2333:00 +CPU= 17.328
  21:50:11 STEP 2334 H=2334:00 +CPU= 27.496
  21:50:28 STEP 2335 H=2335:00 +CPU= 16.797
  21:50:45 STEP 2336 H=2336:00 +CPU= 17.061
  21:51:12 STEP 2337 H=2337:00 +CPU= 26.402
  21:51:30 STEP 2338 H=2338:00 +CPU= 17.975
  21:51:46 STEP 2339 H=2339:00 +CPU= 16.443
  21:52:16 STEP 2340 H=2340:00 +CPU= 30.017
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002340
  21:52:33 STEP 2341 H=2341:00 +CPU= 16.703
  21:52:50 STEP 2342 H=2342:00 +CPU= 17.080
  21:53:16 STEP 2343 H=2343:00 +CPU= 26.437
  21:53:34 STEP 2344 H=2344:00 +CPU= 17.321
  21:53:51 STEP 2345 H=2345:00 +CPU= 17.460
  21:54:18 STEP 2346 H=2346:00 +CPU= 26.581
  21:54:35 STEP 2347 H=2347:00 +CPU= 16.801
  21:54:52 STEP 2348 H=2348:00 +CPU= 17.857
  21:55:19 STEP 2349 H=2349:00 +CPU= 26.198
  21:55:37 STEP 2350 H=2350:00 +CPU= 17.941
  21:55:55 STEP 2351 H=2351:00 +CPU= 18.340
  21:56:28 STEP 2352 H=2352:00 +CPU= 31.514
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_97.zip
Uploading the intermediate file: upload_file_97.zip
  21:56:44 STEP 2353 H=2353:00 +CPU= 16.221
  21:57:02 STEP 2354 H=2354:00 +CPU= 17.482
  21:57:29 STEP 2355 H=2355:00 +CPU= 27.079
  21:57:46 STEP 2356 H=2356:00 +CPU= 17.083
  21:58:02 STEP 2357 H=2357:00 +CPU= 16.498
  21:58:30 STEP 2358 H=2358:00 +CPU= 27.716
  21:58:47 STEP 2359 H=2359:00 +CPU= 16.550
  21:59:03 STEP 2360 H=2360:00 +CPU= 16.863
  21:59:30 STEP 2361 H=2361:00 +CPU= 26.167
  21:59:47 STEP 2362 H=2362:00 +CPU= 17.280
  22:00:04 STEP 2363 H=2363:00 +CPU= 17.166
  22:00:34 STEP 2364 H=2364:00 +CPU= 29.930
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002364
  22:00:51 STEP 2365 H=2365:00 +CPU= 16.847
  22:01:09 STEP 2366 H=2366:00 +CPU= 18.305
  22:01:36 STEP 2367 H=2367:00 +CPU= 26.442
  22:01:53 STEP 2368 H=2368:00 +CPU= 17.724
  22:02:11 STEP 2369 H=2369:00 +CPU= 17.752
  22:02:39 STEP 2370 H=2370:00 +CPU= 28.105
  22:02:56 STEP 2371 H=2371:00 +CPU= 16.571
  22:03:12 STEP 2372 H=2372:00 +CPU= 16.625
  22:03:40 STEP 2373 H=2373:00 +CPU= 27.628
  22:03:57 STEP 2374 H=2374:00 +CPU= 16.800
  22:04:16 STEP 2375 H=2375:00 +CPU= 19.292
  22:04:51 STEP 2376 H=2376:00 +CPU= 33.028
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_98.zip
Uploading the intermediate file: upload_file_98.zip
  22:05:08 STEP 2377 H=2377:00 +CPU= 17.072
Uploading trickle at timestep: 8553600
  22:05:25 STEP 2378 H=2378:00 +CPU= 16.875
  22:05:53 STEP 2379 H=2379:00 +CPU= 28.479
  22:06:12 STEP 2380 H=2380:00 +CPU= 18.463
  22:06:29 STEP 2381 H=2381:00 +CPU= 16.870
  22:06:57 STEP 2382 H=2382:00 +CPU= 28.474
  22:07:16 STEP 2383 H=2383:00 +CPU= 18.329
  22:07:33 STEP 2384 H=2384:00 +CPU= 17.191
  22:08:00 STEP 2385 H=2385:00 +CPU= 27.120
  22:08:18 STEP 2386 H=2386:00 +CPU= 17.790
  22:08:35 STEP 2387 H=2387:00 +CPU= 17.587
  22:09:06 STEP 2388 H=2388:00 +CPU= 30.606
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002388
  22:09:24 STEP 2389 H=2389:00 +CPU= 18.250
  22:09:42 STEP 2390 H=2390:00 +CPU= 18.150
  22:10:10 STEP 2391 H=2391:00 +CPU= 28.126
  22:10:28 STEP 2392 H=2392:00 +CPU= 18.108
  22:10:47 STEP 2393 H=2393:00 +CPU= 18.670
  22:11:17 STEP 2394 H=2394:00 +CPU= 29.817
  22:11:34 STEP 2395 H=2395:00 +CPU= 16.583
  22:11:51 STEP 2396 H=2396:00 +CPU= 17.111
  22:12:19 STEP 2397 H=2397:00 +CPU= 26.552
  22:12:36 STEP 2398 H=2398:00 +CPU= 17.008
  22:12:53 STEP 2399 H=2399:00 +CPU= 16.803
  22:13:26 STEP 2400 H=2400:00 +CPU= 31.503
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_99.zip
Uploading the intermediate file: upload_file_99.zip
  22:13:42 STEP 2401 H=2401:00 +CPU= 16.609
  22:13:59 STEP 2402 H=2402:00 +CPU= 16.708
  22:14:25 STEP 2403 H=2403:00 +CPU= 26.219
  22:14:44 STEP 2404 H=2404:00 +CPU= 18.627
  22:15:00 STEP 2405 H=2405:00 +CPU= 16.186
  22:15:27 STEP 2406 H=2406:00 +CPU= 27.427
  22:15:45 STEP 2407 H=2407:00 +CPU= 17.477
  22:16:01 STEP 2408 H=2408:00 +CPU= 16.674
  22:16:28 STEP 2409 H=2409:00 +CPU= 26.993
  22:16:46 STEP 2410 H=2410:00 +CPU= 17.683
  22:17:03 STEP 2411 H=2411:00 +CPU= 16.407
  22:17:32 STEP 2412 H=2412:00 +CPU= 29.498
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002412
  22:17:49 STEP 2413 H=2413:00 +CPU= 17.413
  22:18:07 STEP 2414 H=2414:00 +CPU= 17.261
  22:18:34 STEP 2415 H=2415:00 +CPU= 27.436
  22:18:52 STEP 2416 H=2416:00 +CPU= 17.887
  22:19:09 STEP 2417 H=2417:00 +CPU= 16.645
  22:19:36 STEP 2418 H=2418:00 +CPU= 27.436
  22:19:54 STEP 2419 H=2419:00 +CPU= 17.748
  22:20:11 STEP 2420 H=2420:00 +CPU= 16.866
  22:20:38 STEP 2421 H=2421:00 +CPU= 27.161
  22:20:55 STEP 2422 H=2422:00 +CPU= 17.234
  22:21:12 STEP 2423 H=2423:00 +CPU= 16.630
  22:22:00 STEP 2424 H=2424:00 +CPU= 32.022
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_100.zip
Uploading the intermediate file: upload_file_100.zip
  22:22:17 STEP 2425 H=2425:00 +CPU= 16.728
  22:22:35 STEP 2426 H=2426:00 +CPU= 18.490
  22:23:03 STEP 2427 H=2427:00 +CPU= 26.453
  22:23:21 STEP 2428 H=2428:00 +CPU= 17.831
  22:23:39 STEP 2429 H=2429:00 +CPU= 17.914
  22:24:08 STEP 2430 H=2430:00 +CPU= 28.950
  22:24:26 STEP 2431 H=2431:00 +CPU= 18.025
  22:24:44 STEP 2432 H=2432:00 +CPU= 18.343
  22:25:12 STEP 2433 H=2433:00 +CPU= 27.953
  22:25:30 STEP 2434 H=2434:00 +CPU= 18.383
  22:25:49 STEP 2435 H=2435:00 +CPU= 18.040
  22:26:19 STEP 2436 H=2436:00 +CPU= 30.025
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002436
  22:26:36 STEP 2437 H=2437:00 +CPU= 17.663
  22:26:53 STEP 2438 H=2438:00 +CPU= 16.938
  22:27:22 STEP 2439 H=2439:00 +CPU= 28.403
  22:27:40 STEP 2440 H=2440:00 +CPU= 17.941
  22:27:56 STEP 2441 H=2441:00 +CPU= 16.588
  22:28:28 STEP 2442 H=2442:00 +CPU= 31.551
  22:28:45 STEP 2443 H=2443:00 +CPU= 17.501
  22:29:03 STEP 2444 H=2444:00 +CPU= 17.403
  22:29:31 STEP 2445 H=2445:00 +CPU= 27.942
  22:29:49 STEP 2446 H=2446:00 +CPU= 18.032
  22:30:06 STEP 2447 H=2447:00 +CPU= 17.659
  22:30:42 STEP 2448 H=2448:00 +CPU= 33.013
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_101.zip
Uploading the intermediate file: upload_file_101.zip
  22:30:59 STEP 2449 H=2449:00 +CPU= 17.925
  22:31:18 STEP 2450 H=2450:00 +CPU= 18.178
  22:31:44 STEP 2451 H=2451:00 +CPU= 26.716
  22:32:02 STEP 2452 H=2452:00 +CPU= 17.834
  22:32:20 STEP 2453 H=2453:00 +CPU= 17.392
  22:32:49 STEP 2454 H=2454:00 +CPU= 29.189
  22:33:07 STEP 2455 H=2455:00 +CPU= 17.650
  22:33:25 STEP 2456 H=2456:00 +CPU= 18.689
  22:33:53 STEP 2457 H=2457:00 +CPU= 27.372
  22:34:10 STEP 2458 H=2458:00 +CPU= 17.376
  22:34:29 STEP 2459 H=2459:00 +CPU= 18.609
  22:34:59 STEP 2460 H=2460:00 +CPU= 30.017
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002460
  22:35:16 STEP 2461 H=2461:00 +CPU= 17.608
  22:35:35 STEP 2462 H=2462:00 +CPU= 19.198
  22:36:05 STEP 2463 H=2463:00 +CPU= 29.311
  22:36:24 STEP 2464 H=2464:00 +CPU= 19.107
  22:36:42 STEP 2465 H=2465:00 +CPU= 18.567
  22:37:10 STEP 2466 H=2466:00 +CPU= 28.071
  22:37:27 STEP 2467 H=2467:00 +CPU= 16.973
  22:37:46 STEP 2468 H=2468:00 +CPU= 18.305
  22:38:14 STEP 2469 H=2469:00 +CPU= 28.449
  22:38:32 STEP 2470 H=2470:00 +CPU= 17.702
  22:38:50 STEP 2471 H=2471:00 +CPU= 18.143
  22:39:25 STEP 2472 H=2472:00 +CPU= 33.695
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_102.zip
Uploading the intermediate file: upload_file_102.zip
  22:39:44 STEP 2473 H=2473:00 +CPU= 18.278
  22:40:01 STEP 2474 H=2474:00 +CPU= 17.476
  22:40:27 STEP 2475 H=2475:00 +CPU= 25.507
  22:40:44 STEP 2476 H=2476:00 +CPU= 17.231
  22:41:00 STEP 2477 H=2477:00 +CPU= 16.441
  22:41:27 STEP 2478 H=2478:00 +CPU= 26.887
  22:41:45 STEP 2479 H=2479:00 +CPU= 17.368
  22:42:02 STEP 2480 H=2480:00 +CPU= 17.309
  22:42:29 STEP 2481 H=2481:00 +CPU= 26.667
  22:42:47 STEP 2482 H=2482:00 +CPU= 17.958
  22:43:03 STEP 2483 H=2483:00 +CPU= 16.399
  22:43:32 STEP 2484 H=2484:00 +CPU= 28.891
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002484
  22:43:49 STEP 2485 H=2485:00 +CPU= 17.282
  22:44:07 STEP 2486 H=2486:00 +CPU= 18.101
  22:44:34 STEP 2487 H=2487:00 +CPU= 26.522
  22:44:52 STEP 2488 H=2488:00 +CPU= 17.881
  22:45:09 STEP 2489 H=2489:00 +CPU= 17.226
  22:45:36 STEP 2490 H=2490:00 +CPU= 27.092
  22:45:53 STEP 2491 H=2491:00 +CPU= 16.980
  22:46:11 STEP 2492 H=2492:00 +CPU= 17.624
  22:46:37 STEP 2493 H=2493:00 +CPU= 25.986
  22:46:54 STEP 2494 H=2494:00 +CPU= 17.488
  22:47:11 STEP 2495 H=2495:00 +CPU= 16.571
  22:47:48 STEP 2496 H=2496:00 +CPU= 31.316
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_103.zip
Uploading the intermediate file: upload_file_103.zip
  22:48:04 STEP 2497 H=2497:00 +CPU= 16.597
  22:48:21 STEP 2498 H=2498:00 +CPU= 17.152
  22:48:48 STEP 2499 H=2499:00 +CPU= 26.791
  22:49:06 STEP 2500 H=2500:00 +CPU= 17.884
  22:49:23 STEP 2501 H=2501:00 +CPU= 17.175
  22:49:51 STEP 2502 H=2502:00 +CPU= 27.541
  22:50:07 STEP 2503 H=2503:00 +CPU= 16.254
  22:50:25 STEP 2504 H=2504:00 +CPU= 17.786
  22:50:51 STEP 2505 H=2505:00 +CPU= 26.596
  22:51:09 STEP 2506 H=2506:00 +CPU= 17.042
  22:51:26 STEP 2507 H=2507:00 +CPU= 17.293
  22:51:55 STEP 2508 H=2508:00 +CPU= 29.305
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002508
  22:52:12 STEP 2509 H=2509:00 +CPU= 16.566
  22:52:30 STEP 2510 H=2510:00 +CPU= 17.794
  22:52:56 STEP 2511 H=2511:00 +CPU= 26.516
  22:53:13 STEP 2512 H=2512:00 +CPU= 17.215
  22:53:31 STEP 2513 H=2513:00 +CPU= 17.372
  22:53:58 STEP 2514 H=2514:00 +CPU= 27.549
  22:54:15 STEP 2515 H=2515:00 +CPU= 17.164
  22:54:33 STEP 2516 H=2516:00 +CPU= 17.469
  22:55:00 STEP 2517 H=2517:00 +CPU= 27.271
  22:55:18 STEP 2518 H=2518:00 +CPU= 17.521
  22:55:36 STEP 2519 H=2519:00 +CPU= 18.505
  22:56:10 STEP 2520 H=2520:00 +CPU= 31.996
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_104.zip
Uploading the intermediate file: upload_file_104.zip
  22:56:28 STEP 2521 H=2521:00 +CPU= 17.137
  22:56:45 STEP 2522 H=2522:00 +CPU= 17.955
  22:57:12 STEP 2523 H=2523:00 +CPU= 26.225
  22:57:29 STEP 2524 H=2524:00 +CPU= 17.269
  22:57:46 STEP 2525 H=2525:00 +CPU= 17.170
  22:58:13 STEP 2526 H=2526:00 +CPU= 26.440
  22:58:29 STEP 2527 H=2527:00 +CPU= 16.539
  22:58:47 STEP 2528 H=2528:00 +CPU= 17.807
  22:59:13 STEP 2529 H=2529:00 +CPU= 26.365
  22:59:31 STEP 2530 H=2530:00 +CPU= 17.708
  22:59:48 STEP 2531 H=2531:00 +CPU= 17.323
  23:00:18 STEP 2532 H=2532:00 +CPU= 29.332
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002532
  23:00:35 STEP 2533 H=2533:00 +CPU= 17.630
  23:00:55 STEP 2534 H=2534:00 +CPU= 19.855
  23:01:21 STEP 2535 H=2535:00 +CPU= 26.136
  23:01:39 STEP 2536 H=2536:00 +CPU= 17.421
  23:01:56 STEP 2537 H=2537:00 +CPU= 17.177
  23:02:23 STEP 2538 H=2538:00 +CPU= 26.693
  23:02:39 STEP 2539 H=2539:00 +CPU= 16.814
  23:02:56 STEP 2540 H=2540:00 +CPU= 17.054
  23:03:22 STEP 2541 H=2541:00 +CPU= 25.839
  23:03:40 STEP 2542 H=2542:00 +CPU= 17.301
  23:03:57 STEP 2543 H=2543:00 +CPU= 17.731
  23:04:32 STEP 2544 H=2544:00 +CPU= 31.566
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_105.zip
Uploading the intermediate file: upload_file_105.zip
  23:04:49 STEP 2545 H=2545:00 +CPU= 16.491
  23:05:06 STEP 2546 H=2546:00 +CPU= 17.457
  23:05:32 STEP 2547 H=2547:00 +CPU= 25.887
  23:05:50 STEP 2548 H=2548:00 +CPU= 17.578
  23:06:08 STEP 2549 H=2549:00 +CPU= 17.987
  23:06:35 STEP 2550 H=2550:00 +CPU= 27.217
  23:06:52 STEP 2551 H=2551:00 +CPU= 17.180
  23:07:09 STEP 2552 H=2552:00 +CPU= 17.299
  23:07:36 STEP 2553 H=2553:00 +CPU= 26.676
  23:07:54 STEP 2554 H=2554:00 +CPU= 17.641
  23:08:11 STEP 2555 H=2555:00 +CPU= 17.379
  23:08:41 STEP 2556 H=2556:00 +CPU= 29.640
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002556
  23:08:58 STEP 2557 H=2557:00 +CPU= 17.142
  23:09:16 STEP 2558 H=2558:00 +CPU= 18.542
  23:09:43 STEP 2559 H=2559:00 +CPU= 27.083
  23:10:01 STEP 2560 H=2560:00 +CPU= 17.389
  23:10:18 STEP 2561 H=2561:00 +CPU= 17.234
  23:10:45 STEP 2562 H=2562:00 +CPU= 26.501
  23:11:01 STEP 2563 H=2563:00 +CPU= 16.552
  23:11:19 STEP 2564 H=2564:00 +CPU= 17.491
  23:11:45 STEP 2565 H=2565:00 +CPU= 26.642
  23:12:03 STEP 2566 H=2566:00 +CPU= 17.824
  23:12:21 STEP 2567 H=2567:00 +CPU= 17.793
  23:12:57 STEP 2568 H=2568:00 +CPU= 31.702
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_106.zip
Uploading the intermediate file: upload_file_106.zip
  23:13:13 STEP 2569 H=2569:00 +CPU= 16.599
  23:13:31 STEP 2570 H=2570:00 +CPU= 17.395
  23:13:57 STEP 2571 H=2571:00 +CPU= 26.890
  23:14:14 STEP 2572 H=2572:00 +CPU= 16.831
  23:14:31 STEP 2573 H=2573:00 +CPU= 16.765
  23:14:59 STEP 2574 H=2574:00 +CPU= 27.889
  23:15:16 STEP 2575 H=2575:00 +CPU= 17.012
  23:15:34 STEP 2576 H=2576:00 +CPU= 17.581
  23:16:00 STEP 2577 H=2577:00 +CPU= 26.554
  23:16:17 STEP 2578 H=2578:00 +CPU= 17.314
  23:16:34 STEP 2579 H=2579:00 +CPU= 16.636
  23:17:04 STEP 2580 H=2580:00 +CPU= 29.704
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002580
  23:17:21 STEP 2581 H=2581:00 +CPU= 16.736
  23:17:38 STEP 2582 H=2582:00 +CPU= 17.299
  23:18:05 STEP 2583 H=2583:00 +CPU= 27.151
  23:18:23 STEP 2584 H=2584:00 +CPU= 17.710
  23:18:40 STEP 2585 H=2585:00 +CPU= 17.111
  23:19:06 STEP 2586 H=2586:00 +CPU= 26.639
  23:19:24 STEP 2587 H=2587:00 +CPU= 17.034
  23:19:41 STEP 2588 H=2588:00 +CPU= 17.788
  23:20:08 STEP 2589 H=2589:00 +CPU= 27.020
  23:20:25 STEP 2590 H=2590:00 +CPU= 17.154
  23:20:43 STEP 2591 H=2591:00 +CPU= 16.942
  23:21:20 STEP 2592 H=2592:00 +CPU= 31.188
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_107.zip
Uploading the intermediate file: upload_file_107.zip
  23:21:36 STEP 2593 H=2593:00 +CPU= 16.447
  23:21:53 STEP 2594 H=2594:00 +CPU= 16.985
  23:22:20 STEP 2595 H=2595:00 +CPU= 26.443
  23:22:37 STEP 2596 H=2596:00 +CPU= 17.465
  23:22:54 STEP 2597 H=2597:00 +CPU= 16.752
  23:23:22 STEP 2598 H=2598:00 +CPU= 27.902
  23:23:39 STEP 2599 H=2599:00 +CPU= 17.389
  23:23:56 STEP 2600 H=2600:00 +CPU= 17.007
  23:24:23 STEP 2601 H=2601:00 +CPU= 26.949
  23:24:41 STEP 2602 H=2602:00 +CPU= 17.809
  23:24:59 STEP 2603 H=2603:00 +CPU= 17.497
  23:25:27 STEP 2604 H=2604:00 +CPU= 28.737
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002604
  23:25:44 STEP 2605 H=2605:00 +CPU= 16.789
  23:26:02 STEP 2606 H=2606:00 +CPU= 17.622
  23:26:28 STEP 2607 H=2607:00 +CPU= 26.241
  23:26:45 STEP 2608 H=2608:00 +CPU= 17.171
  23:27:02 STEP 2609 H=2609:00 +CPU= 16.955
  23:27:30 STEP 2610 H=2610:00 +CPU= 27.235
  23:27:46 STEP 2611 H=2611:00 +CPU= 16.901
  23:28:04 STEP 2612 H=2612:00 +CPU= 17.770
  23:28:31 STEP 2613 H=2613:00 +CPU= 26.549
  23:28:49 STEP 2614 H=2614:00 +CPU= 17.853
  23:29:06 STEP 2615 H=2615:00 +CPU= 17.679
  23:29:47 STEP 2616 H=2616:00 +CPU= 31.257
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_108.zip
Uploading the intermediate file: upload_file_108.zip
  23:30:04 STEP 2617 H=2617:00 +CPU= 17.271
Uploading trickle at timestep: 9417600
  23:30:21 STEP 2618 H=2618:00 +CPU= 17.426
  23:30:49 STEP 2619 H=2619:00 +CPU= 27.328
  23:31:07 STEP 2620 H=2620:00 +CPU= 18.158
  23:31:24 STEP 2621 H=2621:00 +CPU= 17.232
  23:31:51 STEP 2622 H=2622:00 +CPU= 27.494
  23:32:09 STEP 2623 H=2623:00 +CPU= 17.134
  23:32:26 STEP 2624 H=2624:00 +CPU= 17.403
  23:32:53 STEP 2625 H=2625:00 +CPU= 26.903
  23:33:10 STEP 2626 H=2626:00 +CPU= 17.369
  23:33:27 STEP 2627 H=2627:00 +CPU= 16.505
  23:33:56 STEP 2628 H=2628:00 +CPU= 29.044
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002628
  23:34:13 STEP 2629 H=2629:00 +CPU= 17.296
  23:34:30 STEP 2630 H=2630:00 +CPU= 17.297
  23:34:56 STEP 2631 H=2631:00 +CPU= 25.651
  23:35:13 STEP 2632 H=2632:00 +CPU= 17.355
  23:35:30 STEP 2633 H=2633:00 +CPU= 16.494
  23:35:56 STEP 2634 H=2634:00 +CPU= 26.002
  23:36:12 STEP 2635 H=2635:00 +CPU= 16.372
  23:36:30 STEP 2636 H=2636:00 +CPU= 17.530
  23:36:56 STEP 2637 H=2637:00 +CPU= 25.638
  23:37:13 STEP 2638 H=2638:00 +CPU= 17.404
  23:37:30 STEP 2639 H=2639:00 +CPU= 16.993
  23:38:07 STEP 2640 H=2640:00 +CPU= 31.132
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_109.zip
Uploading the intermediate file: upload_file_109.zip
  23:38:23 STEP 2641 H=2641:00 +CPU= 16.520
  23:38:40 STEP 2642 H=2642:00 +CPU= 17.058
  23:39:07 STEP 2643 H=2643:00 +CPU= 26.303
  23:39:24 STEP 2644 H=2644:00 +CPU= 17.341
  23:39:40 STEP 2645 H=2645:00 +CPU= 16.183
  23:40:07 STEP 2646 H=2646:00 +CPU= 26.776
  23:40:24 STEP 2647 H=2647:00 +CPU= 16.816
  23:40:41 STEP 2648 H=2648:00 +CPU= 16.844
  23:41:07 STEP 2649 H=2649:00 +CPU= 25.985
  23:41:24 STEP 2650 H=2650:00 +CPU= 16.889
  23:41:40 STEP 2651 H=2651:00 +CPU= 16.789
  23:42:09 STEP 2652 H=2652:00 +CPU= 28.870
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002652
  23:42:26 STEP 2653 H=2653:00 +CPU= 16.856
  23:42:43 STEP 2654 H=2654:00 +CPU= 17.266
  23:43:09 STEP 2655 H=2655:00 +CPU= 25.830
  23:43:26 STEP 2656 H=2656:00 +CPU= 17.063
  23:43:43 STEP 2657 H=2657:00 +CPU= 17.115
  23:44:11 STEP 2658 H=2658:00 +CPU= 27.364
  23:44:27 STEP 2659 H=2659:00 +CPU= 16.688
  23:44:45 STEP 2660 H=2660:00 +CPU= 17.139
  23:45:11 STEP 2661 H=2661:00 +CPU= 25.987
  23:45:27 STEP 2662 H=2662:00 +CPU= 16.901
  23:45:46 STEP 2663 H=2663:00 +CPU= 18.512
  23:46:22 STEP 2664 H=2664:00 +CPU= 31.220
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_110.zip
Uploading the intermediate file: upload_file_110.zip
  23:46:39 STEP 2665 H=2665:00 +CPU= 16.793
  23:46:56 STEP 2666 H=2666:00 +CPU= 17.343
  23:47:22 STEP 2667 H=2667:00 +CPU= 26.085
  23:47:40 STEP 2668 H=2668:00 +CPU= 17.901
  23:47:58 STEP 2669 H=2669:00 +CPU= 17.444
  23:48:24 STEP 2670 H=2670:00 +CPU= 26.927
  23:48:42 STEP 2671 H=2671:00 +CPU= 17.775
  23:49:00 STEP 2672 H=2672:00 +CPU= 17.362
  23:49:26 STEP 2673 H=2673:00 +CPU= 26.620
  23:49:44 STEP 2674 H=2674:00 +CPU= 17.571
  23:50:00 STEP 2675 H=2675:00 +CPU= 16.647
  23:50:30 STEP 2676 H=2676:00 +CPU= 29.686
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002676
  23:50:48 STEP 2677 H=2677:00 +CPU= 17.521
  23:51:06 STEP 2678 H=2678:00 +CPU= 17.926
  23:51:32 STEP 2679 H=2679:00 +CPU= 26.456
  23:51:50 STEP 2680 H=2680:00 +CPU= 17.510
  23:52:06 STEP 2681 H=2681:00 +CPU= 16.567
  23:52:34 STEP 2682 H=2682:00 +CPU= 27.674
  23:52:51 STEP 2683 H=2683:00 +CPU= 16.856
  23:53:08 STEP 2684 H=2684:00 +CPU= 17.279
  23:53:33 STEP 2685 H=2685:00 +CPU= 25.554
  23:53:51 STEP 2686 H=2686:00 +CPU= 17.664
  23:54:08 STEP 2687 H=2687:00 +CPU= 17.137
  23:54:59 STEP 2688 H=2688:00 +CPU= 31.480
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_111.zip
Uploading the intermediate file: upload_file_111.zip
  23:55:15 STEP 2689 H=2689:00 +CPU= 16.049
  23:55:32 STEP 2690 H=2690:00 +CPU= 17.064
  23:55:58 STEP 2691 H=2691:00 +CPU= 26.000
  23:56:16 STEP 2692 H=2692:00 +CPU= 17.682
  23:56:33 STEP 2693 H=2693:00 +CPU= 17.058
  23:57:00 STEP 2694 H=2694:00 +CPU= 27.307
  23:57:16 STEP 2695 H=2695:00 +CPU= 16.168
  23:57:33 STEP 2696 H=2696:00 +CPU= 16.837
  23:58:00 STEP 2697 H=2697:00 +CPU= 26.386
  23:58:16 STEP 2698 H=2698:00 +CPU= 16.772
  23:58:33 STEP 2699 H=2699:00 +CPU= 16.459
  23:59:02 STEP 2700 H=2700:00 +CPU= 29.348
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002700
  23:59:19 STEP 2701 H=2701:00 +CPU= 16.674
  23:59:36 STEP 2702 H=2702:00 +CPU= 16.926
  00:00:03 STEP 2703 H=2703:00 +CPU= 27.546
  00:00:21 STEP 2704 H=2704:00 +CPU= 17.590
  00:00:37 STEP 2705 H=2705:00 +CPU= 16.377
  00:01:04 STEP 2706 H=2706:00 +CPU= 26.310
  00:01:21 STEP 2707 H=2707:00 +CPU= 17.082
  00:01:37 STEP 2708 H=2708:00 +CPU= 16.522
  00:02:03 STEP 2709 H=2709:00 +CPU= 25.778
  00:02:20 STEP 2710 H=2710:00 +CPU= 16.983
  00:02:37 STEP 2711 H=2711:00 +CPU= 16.817
  00:03:12 STEP 2712 H=2712:00 +CPU= 31.291
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_112.zip
Uploading the intermediate file: upload_file_112.zip
  00:03:28 STEP 2713 H=2713:00 +CPU= 15.761
  00:03:45 STEP 2714 H=2714:00 +CPU= 17.165
  00:04:11 STEP 2715 H=2715:00 +CPU= 25.551
  00:04:28 STEP 2716 H=2716:00 +CPU= 17.220
  00:04:44 STEP 2717 H=2717:00 +CPU= 15.757
  00:05:11 STEP 2718 H=2718:00 +CPU= 27.204
  00:05:28 STEP 2719 H=2719:00 +CPU= 17.067
  00:05:45 STEP 2720 H=2720:00 +CPU= 16.469
  00:06:11 STEP 2721 H=2721:00 +CPU= 26.574
  00:06:29 STEP 2722 H=2722:00 +CPU= 17.896
  00:06:45 STEP 2723 H=2723:00 +CPU= 16.160
  00:07:14 STEP 2724 H=2724:00 +CPU= 28.702
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002724
  00:07:32 STEP 2725 H=2725:00 +CPU= 17.575
  00:07:48 STEP 2726 H=2726:00 +CPU= 16.734
  00:08:14 STEP 2727 H=2727:00 +CPU= 25.462
  00:08:32 STEP 2728 H=2728:00 +CPU= 17.957
  00:08:48 STEP 2729 H=2729:00 +CPU= 16.387
  00:09:15 STEP 2730 H=2730:00 +CPU= 26.818
  00:09:33 STEP 2731 H=2731:00 +CPU= 17.746
  00:09:50 STEP 2732 H=2732:00 +CPU= 17.226
  00:10:16 STEP 2733 H=2733:00 +CPU= 25.766
  00:10:33 STEP 2734 H=2734:00 +CPU= 17.338
  00:10:49 STEP 2735 H=2735:00 +CPU= 16.370
  00:11:21 STEP 2736 H=2736:00 +CPU= 30.761
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_113.zip
Uploading the intermediate file: upload_file_113.zip
  00:11:37 STEP 2737 H=2737:00 +CPU= 15.837
  00:11:54 STEP 2738 H=2738:00 +CPU= 17.325
  00:12:21 STEP 2739 H=2739:00 +CPU= 26.513
  00:12:38 STEP 2740 H=2740:00 +CPU= 17.657
  00:12:54 STEP 2741 H=2741:00 +CPU= 16.038
  00:13:21 STEP 2742 H=2742:00 +CPU= 26.878
  00:13:38 STEP 2743 H=2743:00 +CPU= 16.780
  00:13:55 STEP 2744 H=2744:00 +CPU= 16.516
  00:14:20 STEP 2745 H=2745:00 +CPU= 25.510
  00:14:38 STEP 2746 H=2746:00 +CPU= 17.982
  00:14:54 STEP 2747 H=2747:00 +CPU= 16.039
  00:15:24 STEP 2748 H=2748:00 +CPU= 29.441
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002748
  00:15:41 STEP 2749 H=2749:00 +CPU= 17.191
  00:15:59 STEP 2750 H=2750:00 +CPU= 18.373
  00:16:27 STEP 2751 H=2751:00 +CPU= 27.328
  00:16:43 STEP 2752 H=2752:00 +CPU= 16.392
  00:17:00 STEP 2753 H=2753:00 +CPU= 16.640
  00:17:27 STEP 2754 H=2754:00 +CPU= 27.077
  00:17:43 STEP 2755 H=2755:00 +CPU= 16.043
  00:18:00 STEP 2756 H=2756:00 +CPU= 16.928
  00:18:26 STEP 2757 H=2757:00 +CPU= 26.171
  00:18:43 STEP 2758 H=2758:00 +CPU= 16.736
  00:18:59 STEP 2759 H=2759:00 +CPU= 16.516
  00:19:32 STEP 2760 H=2760:00 +CPU= 31.823
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_114.zip
Uploading the intermediate file: upload_file_114.zip
  00:19:48 STEP 2761 H=2761:00 +CPU= 16.076
  00:20:05 STEP 2762 H=2762:00 +CPU= 17.085
  00:20:31 STEP 2763 H=2763:00 +CPU= 25.337
  00:20:48 STEP 2764 H=2764:00 +CPU= 17.065
  00:21:04 STEP 2765 H=2765:00 +CPU= 16.395
  00:21:31 STEP 2766 H=2766:00 +CPU= 26.675
  00:21:48 STEP 2767 H=2767:00 +CPU= 17.600
  00:22:05 STEP 2768 H=2768:00 +CPU= 16.832
  00:22:33 STEP 2769 H=2769:00 +CPU= 27.720
  00:22:51 STEP 2770 H=2770:00 +CPU= 18.371
  00:23:09 STEP 2771 H=2771:00 +CPU= 17.527
  00:23:39 STEP 2772 H=2772:00 +CPU= 30.124
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002772
  00:23:57 STEP 2773 H=2773:00 +CPU= 17.662
  00:24:15 STEP 2774 H=2774:00 +CPU= 18.152
  00:24:44 STEP 2775 H=2775:00 +CPU= 28.739
  00:25:01 STEP 2776 H=2776:00 +CPU= 17.502
  00:25:18 STEP 2777 H=2777:00 +CPU= 16.715
  00:25:45 STEP 2778 H=2778:00 +CPU= 27.096
  00:26:02 STEP 2779 H=2779:00 +CPU= 16.693
  00:26:19 STEP 2780 H=2780:00 +CPU= 17.361
  00:26:46 STEP 2781 H=2781:00 +CPU= 27.413
  00:27:04 STEP 2782 H=2782:00 +CPU= 17.433
  00:27:21 STEP 2783 H=2783:00 +CPU= 17.097
  00:27:54 STEP 2784 H=2784:00 +CPU= 31.668
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_115.zip
Uploading the intermediate file: upload_file_115.zip
  00:28:11 STEP 2785 H=2785:00 +CPU= 17.109
  00:28:28 STEP 2786 H=2786:00 +CPU= 17.198
  00:28:56 STEP 2787 H=2787:00 +CPU= 28.529
  00:29:14 STEP 2788 H=2788:00 +CPU= 17.849
  00:29:32 STEP 2789 H=2789:00 +CPU= 17.851
  00:29:59 STEP 2790 H=2790:00 +CPU= 26.827
  00:30:15 STEP 2791 H=2791:00 +CPU= 16.453
  00:30:33 STEP 2792 H=2792:00 +CPU= 17.160
  00:30:58 STEP 2793 H=2793:00 +CPU= 25.664
  00:31:15 STEP 2794 H=2794:00 +CPU= 16.662
  00:31:32 STEP 2795 H=2795:00 +CPU= 16.693
  00:31:59 STEP 2796 H=2796:00 +CPU= 27.804
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002796
  00:32:16 STEP 2797 H=2797:00 +CPU= 16.506
  00:32:33 STEP 2798 H=2798:00 +CPU= 16.954
  00:32:58 STEP 2799 H=2799:00 +CPU= 25.307
  00:33:15 STEP 2800 H=2800:00 +CPU= 17.213
  00:33:32 STEP 2801 H=2801:00 +CPU= 16.626
  00:33:59 STEP 2802 H=2802:00 +CPU= 26.847
  00:34:15 STEP 2803 H=2803:00 +CPU= 16.466
  00:34:33 STEP 2804 H=2804:00 +CPU= 17.681
  00:34:59 STEP 2805 H=2805:00 +CPU= 26.302
  00:35:17 STEP 2806 H=2806:00 +CPU= 17.608
  00:35:33 STEP 2807 H=2807:00 +CPU= 16.299
  00:36:07 STEP 2808 H=2808:00 +CPU= 30.416
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_116.zip
Uploading the intermediate file: upload_file_116.zip
  00:36:23 STEP 2809 H=2809:00 +CPU= 16.387
  00:36:40 STEP 2810 H=2810:00 +CPU= 17.299
  00:37:05 STEP 2811 H=2811:00 +CPU= 24.929
  00:37:23 STEP 2812 H=2812:00 +CPU= 17.526
  00:37:39 STEP 2813 H=2813:00 +CPU= 16.226
  00:38:06 STEP 2814 H=2814:00 +CPU= 26.952
  00:38:22 STEP 2815 H=2815:00 +CPU= 16.178
  00:38:39 STEP 2816 H=2816:00 +CPU= 16.764
  00:39:05 STEP 2817 H=2817:00 +CPU= 25.884
  00:39:22 STEP 2818 H=2818:00 +CPU= 17.148
  00:39:38 STEP 2819 H=2819:00 +CPU= 16.009
  00:40:08 STEP 2820 H=2820:00 +CPU= 29.642
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002820
  00:40:24 STEP 2821 H=2821:00 +CPU= 16.545
  00:40:42 STEP 2822 H=2822:00 +CPU= 17.511
  00:41:08 STEP 2823 H=2823:00 +CPU= 26.753
  00:41:26 STEP 2824 H=2824:00 +CPU= 17.255
  00:41:42 STEP 2825 H=2825:00 +CPU= 16.155
  00:42:08 STEP 2826 H=2826:00 +CPU= 26.497
  00:42:25 STEP 2827 H=2827:00 +CPU= 16.462
  00:42:41 STEP 2828 H=2828:00 +CPU= 16.500
  00:43:07 STEP 2829 H=2829:00 +CPU= 26.229
  00:43:26 STEP 2830 H=2830:00 +CPU= 18.552
  00:43:43 STEP 2831 H=2831:00 +CPU= 16.580
  00:44:14 STEP 2832 H=2832:00 +CPU= 30.562
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_117.zip
Uploading the intermediate file: upload_file_117.zip
  00:44:32 STEP 2833 H=2833:00 +CPU= 18.344
  00:44:49 STEP 2834 H=2834:00 +CPU= 16.942
  00:45:15 STEP 2835 H=2835:00 +CPU= 26.138
  00:45:33 STEP 2836 H=2836:00 +CPU= 17.066
  00:45:49 STEP 2837 H=2837:00 +CPU= 16.072
  00:46:16 STEP 2838 H=2838:00 +CPU= 27.053
  00:46:34 STEP 2839 H=2839:00 +CPU= 18.050
  00:46:51 STEP 2840 H=2840:00 +CPU= 17.074
  00:47:17 STEP 2841 H=2841:00 +CPU= 26.204
  00:47:36 STEP 2842 H=2842:00 +CPU= 18.706
  00:47:52 STEP 2843 H=2843:00 +CPU= 16.494
  00:48:21 STEP 2844 H=2844:00 +CPU= 29.009
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002844
  00:48:38 STEP 2845 H=2845:00 +CPU= 16.413
  00:48:55 STEP 2846 H=2846:00 +CPU= 17.466
  00:49:22 STEP 2847 H=2847:00 +CPU= 27.264
  00:49:39 STEP 2848 H=2848:00 +CPU= 17.062
  00:49:56 STEP 2849 H=2849:00 +CPU= 16.358
  00:50:22 STEP 2850 H=2850:00 +CPU= 26.563
  00:50:40 STEP 2851 H=2851:00 +CPU= 17.706
  00:50:57 STEP 2852 H=2852:00 +CPU= 16.925
  00:51:23 STEP 2853 H=2853:00 +CPU= 26.348
  00:51:41 STEP 2854 H=2854:00 +CPU= 17.248
  00:51:57 STEP 2855 H=2855:00 +CPU= 16.227
  00:52:29 STEP 2856 H=2856:00 +CPU= 31.390
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_118.zip
Uploading the intermediate file: upload_file_118.zip
  00:52:45 STEP 2857 H=2857:00 +CPU= 16.564
Uploading trickle at timestep: 10281600
  00:53:03 STEP 2858 H=2858:00 +CPU= 17.882
  00:53:29 STEP 2859 H=2859:00 +CPU= 26.218
  00:53:46 STEP 2860 H=2860:00 +CPU= 16.489
  00:54:02 STEP 2861 H=2861:00 +CPU= 16.104
  00:54:29 STEP 2862 H=2862:00 +CPU= 27.177
  00:54:46 STEP 2863 H=2863:00 +CPU= 16.392
  00:55:03 STEP 2864 H=2864:00 +CPU= 16.887
  00:55:28 STEP 2865 H=2865:00 +CPU= 25.324
  00:55:47 STEP 2866 H=2866:00 +CPU= 18.763
  00:56:03 STEP 2867 H=2867:00 +CPU= 16.022
  00:56:31 STEP 2868 H=2868:00 +CPU= 28.760
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002868
  00:56:50 STEP 2869 H=2869:00 +CPU= 18.128
  00:57:07 STEP 2870 H=2870:00 +CPU= 17.038
  00:57:33 STEP 2871 H=2871:00 +CPU= 25.962
  00:57:50 STEP 2872 H=2872:00 +CPU= 17.614
  00:58:06 STEP 2873 H=2873:00 +CPU= 15.803
  00:58:32 STEP 2874 H=2874:00 +CPU= 25.770
  00:58:49 STEP 2875 H=2875:00 +CPU= 17.102
  00:59:06 STEP 2876 H=2876:00 +CPU= 16.695
  00:59:32 STEP 2877 H=2877:00 +CPU= 26.330
  00:59:49 STEP 2878 H=2878:00 +CPU= 17.305
  01:00:06 STEP 2879 H=2879:00 +CPU= 16.561
  01:00:39 STEP 2880 H=2880:00 +CPU= 30.545
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_119.zip
Uploading the intermediate file: upload_file_119.zip
  01:00:56 STEP 2881 H=2881:00 +CPU= 17.023
  01:01:13 STEP 2882 H=2882:00 +CPU= 16.722
  01:01:40 STEP 2883 H=2883:00 +CPU= 27.813
  01:01:58 STEP 2884 H=2884:00 +CPU= 17.303
  01:02:14 STEP 2885 H=2885:00 +CPU= 16.512
  01:02:42 STEP 2886 H=2886:00 +CPU= 27.762
  01:02:59 STEP 2887 H=2887:00 +CPU= 16.925
  01:03:17 STEP 2888 H=2888:00 +CPU= 17.623
  01:03:45 STEP 2889 H=2889:00 +CPU= 28.263
  01:04:03 STEP 2890 H=2890:00 +CPU= 18.094
  01:04:21 STEP 2891 H=2891:00 +CPU= 17.926
  01:04:52 STEP 2892 H=2892:00 +CPU= 31.178
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002892
  01:05:09 STEP 2893 H=2893:00 +CPU= 16.857
  01:05:26 STEP 2894 H=2894:00 +CPU= 16.624
  01:05:52 STEP 2895 H=2895:00 +CPU= 26.551
  01:06:10 STEP 2896 H=2896:00 +CPU= 17.537
  01:06:26 STEP 2897 H=2897:00 +CPU= 16.728
  01:06:54 STEP 2898 H=2898:00 +CPU= 27.363
  01:07:11 STEP 2899 H=2899:00 +CPU= 17.418
  01:07:29 STEP 2900 H=2900:00 +CPU= 18.197
  01:07:59 STEP 2901 H=2901:00 +CPU= 29.480
  01:08:17 STEP 2902 H=2902:00 +CPU= 17.618
  01:08:34 STEP 2903 H=2903:00 +CPU= 17.547
  01:09:08 STEP 2904 H=2904:00 +CPU= 32.070
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_120.zip
Uploading the intermediate file: upload_file_120.zip
  01:09:25 STEP 2905 H=2905:00 +CPU= 16.810
  01:09:42 STEP 2906 H=2906:00 +CPU= 17.280
  01:10:09 STEP 2907 H=2907:00 +CPU= 26.552
  01:10:26 STEP 2908 H=2908:00 +CPU= 17.227
  01:10:42 STEP 2909 H=2909:00 +CPU= 15.886
  01:11:10 STEP 2910 H=2910:00 +CPU= 27.612
  01:11:26 STEP 2911 H=2911:00 +CPU= 16.783
  01:11:44 STEP 2912 H=2912:00 +CPU= 17.433
  01:12:14 STEP 2913 H=2913:00 +CPU= 30.059
  01:12:33 STEP 2914 H=2914:00 +CPU= 18.516
  01:12:49 STEP 2915 H=2915:00 +CPU= 16.370
  01:13:19 STEP 2916 H=2916:00 +CPU= 30.075
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002916
  01:13:36 STEP 2917 H=2917:00 +CPU= 17.095
  01:13:53 STEP 2918 H=2918:00 +CPU= 16.740
  01:14:20 STEP 2919 H=2919:00 +CPU= 26.997
  01:14:36 STEP 2920 H=2920:00 +CPU= 16.587
  01:14:53 STEP 2921 H=2921:00 +CPU= 17.059
  01:15:22 STEP 2922 H=2922:00 +CPU= 28.679
  01:15:39 STEP 2923 H=2923:00 +CPU= 17.181
  01:15:58 STEP 2924 H=2924:00 +CPU= 18.624
  01:16:27 STEP 2925 H=2925:00 +CPU= 28.916
  01:16:44 STEP 2926 H=2926:00 +CPU= 17.461
  01:17:03 STEP 2927 H=2927:00 +CPU= 18.348
  01:17:35 STEP 2928 H=2928:00 +CPU= 30.782
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_121.zip
Uploading the intermediate file: upload_file_121.zip
  01:17:52 STEP 2929 H=2929:00 +CPU= 17.235
  01:18:10 STEP 2930 H=2930:00 +CPU= 17.887
  01:18:38 STEP 2931 H=2931:00 +CPU= 27.717
  01:18:56 STEP 2932 H=2932:00 +CPU= 18.554
  01:19:13 STEP 2933 H=2933:00 +CPU= 16.601
  01:19:40 STEP 2934 H=2934:00 +CPU= 26.853
  01:19:56 STEP 2935 H=2935:00 +CPU= 16.724
  01:20:14 STEP 2936 H=2936:00 +CPU= 17.259
  01:20:40 STEP 2937 H=2937:00 +CPU= 26.089
  01:20:56 STEP 2938 H=2938:00 +CPU= 16.820
  01:21:13 STEP 2939 H=2939:00 +CPU= 16.418
  01:21:42 STEP 2940 H=2940:00 +CPU= 29.522
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002940
  01:22:00 STEP 2941 H=2941:00 +CPU= 17.187
  01:22:17 STEP 2942 H=2942:00 +CPU= 17.580
  01:22:44 STEP 2943 H=2943:00 +CPU= 26.340
  01:23:01 STEP 2944 H=2944:00 +CPU= 17.320
  01:23:17 STEP 2945 H=2945:00 +CPU= 16.416
  01:23:44 STEP 2946 H=2946:00 +CPU= 26.909
  01:24:01 STEP 2947 H=2947:00 +CPU= 16.785
  01:24:18 STEP 2948 H=2948:00 +CPU= 16.672
  01:24:45 STEP 2949 H=2949:00 +CPU= 26.888
  01:25:01 STEP 2950 H=2950:00 +CPU= 16.687
  01:25:18 STEP 2951 H=2951:00 +CPU= 16.873
  01:25:50 STEP 2952 H=2952:00 +CPU= 30.905
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/0/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/0/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12190811/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0167_2005050100_123_974_12190811_0_r1894542908_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
01:29:13 (51): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:05 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 10,623,600 45,747 0.0043
09 May 2023 16:09:05 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 10,281,600 43,762 0.0043
09 May 2023 16:08:58 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 9,417,600 38,838 0.0041
09 May 2023 16:08:54 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 8,553,600 33,793 0.0040
09 May 2023 16:08:50 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 7,689,600 28,751 0.0037
09 May 2023 16:08:50 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 7,689,600 28,751 0.0037
09 May 2023 16:08:43 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 6,825,600 23,735 0.0035
09 May 2023 16:08:43 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 5,961,600 20,887 0.0035
09 May 2023 16:08:43 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 5,961,600 20,887 0.0035
09 May 2023 16:08:39 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 5,097,600 17,974 0.0035
09 May 2023 16:08:36 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 4,233,600 15,158 0.0036
09 May 2023 16:08:27 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 2,592,000 9,317 0.0036
09 May 2023 16:08:24 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 1,728,000 6,367 0.0037
09 May 2023 16:08:24 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 1,728,000 6,367 0.0037
09 May 2023 16:08:21 1537625 22276464 oifs_43r3_ps_0167_2005050100_123_974_12190811_0 864,000 2,990 0.0035


©2024 climateprediction.net