climateprediction.net home page
Task 22273057

Task 22273057

Name oifs_43r3_ps_0977_2001050100_123_970_12187621_0
Workunit 12187621
Created 23 Dec 2022, 1:31:05 UTC
Sent 5 Jan 2023, 1:08:55 UTC
Report deadline 4 Feb 2023, 1:08:55 UTC
Received 14 Jan 2023, 23:55:14 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1537625
Run time 13 hours 49 min 47 sec
CPU time 13 hours 35 min 52 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,439.09 MB
Peak swap size 4,977.85 MB
Peak disk usage 4,244.08 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<stderr_txt>
21/ICMSHhq0f+002268
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002268
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_94.zip
Uploading the intermediate file: upload_file_94.zip
  23:31:16 STEP 2281 H=2281:00 +CPU= 17.043
  23:31:33 STEP 2282 H=2282:00 +CPU= 17.103
  23:32:00 STEP 2283 H=2283:00 +CPU= 26.725
  23:32:20 STEP 2284 H=2284:00 +CPU= 20.179
  23:32:38 STEP 2285 H=2285:00 +CPU= 17.807
  23:33:06 STEP 2286 H=2286:00 +CPU= 27.871
  23:33:22 STEP 2287 H=2287:00 +CPU= 16.515
  23:33:39 STEP 2288 H=2288:00 +CPU= 17.088
  23:34:07 STEP 2289 H=2289:00 +CPU= 27.457
  23:34:24 STEP 2290 H=2290:00 +CPU= 17.186
  23:34:41 STEP 2291 H=2291:00 +CPU= 16.527
  23:35:10 STEP 2292 H=2292:00 +CPU= 29.338
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002292
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002292
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002292
  23:35:27 STEP 2293 H=2293:00 +CPU= 17.033
  23:35:44 STEP 2294 H=2294:00 +CPU= 17.405
  23:36:10 STEP 2295 H=2295:00 +CPU= 25.884
  23:36:28 STEP 2296 H=2296:00 +CPU= 17.769
  23:36:45 STEP 2297 H=2297:00 +CPU= 16.882
  23:37:12 STEP 2298 H=2298:00 +CPU= 27.427
  23:37:29 STEP 2299 H=2299:00 +CPU= 16.334
  23:37:46 STEP 2300 H=2300:00 +CPU= 17.140
  23:38:12 STEP 2301 H=2301:00 +CPU= 25.928
  23:38:30 STEP 2302 H=2302:00 +CPU= 18.264
  23:38:46 STEP 2303 H=2303:00 +CPU= 16.405
  23:39:20 STEP 2304 H=2304:00 +CPU= 31.976
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002304
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002304
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002280
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002292
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002292
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_95.zip
Uploading the intermediate file: upload_file_95.zip
  23:39:36 STEP 2305 H=2305:00 +CPU= 16.385
Uploading trickle at timestep: 8294400
  23:39:53 STEP 2306 H=2306:00 +CPU= 17.276
  23:40:19 STEP 2307 H=2307:00 +CPU= 25.685
  23:40:37 STEP 2308 H=2308:00 +CPU= 17.444
  23:40:53 STEP 2309 H=2309:00 +CPU= 16.409
  23:41:19 STEP 2310 H=2310:00 +CPU= 26.344
  23:41:36 STEP 2311 H=2311:00 +CPU= 16.937
  23:41:53 STEP 2312 H=2312:00 +CPU= 17.256
  23:42:19 STEP 2313 H=2313:00 +CPU= 25.780
  23:42:37 STEP 2314 H=2314:00 +CPU= 17.759
  23:42:53 STEP 2315 H=2315:00 +CPU= 16.213
  23:43:23 STEP 2316 H=2316:00 +CPU= 29.267
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002316
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002316
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002316
  23:43:39 STEP 2317 H=2317:00 +CPU= 16.730
  23:43:56 STEP 2318 H=2318:00 +CPU= 16.996
  23:44:24 STEP 2319 H=2319:00 +CPU= 27.411
  23:44:41 STEP 2320 H=2320:00 +CPU= 17.404
  23:44:58 STEP 2321 H=2321:00 +CPU= 16.633
  23:45:24 STEP 2322 H=2322:00 +CPU= 26.490
  23:45:41 STEP 2323 H=2323:00 +CPU= 16.892
  23:46:00 STEP 2324 H=2324:00 +CPU= 18.318
  23:46:27 STEP 2325 H=2325:00 +CPU= 27.221
  23:46:44 STEP 2326 H=2326:00 +CPU= 17.594
  23:47:01 STEP 2327 H=2327:00 +CPU= 17.128
  23:47:34 STEP 2328 H=2328:00 +CPU= 31.719
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002328
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002328
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002304
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002316
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002316
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_96.zip
Uploading the intermediate file: upload_file_96.zip
  23:47:51 STEP 2329 H=2329:00 +CPU= 16.949
  23:48:09 STEP 2330 H=2330:00 +CPU= 17.770
  23:48:36 STEP 2331 H=2331:00 +CPU= 27.398
  23:48:54 STEP 2332 H=2332:00 +CPU= 17.427
  23:49:11 STEP 2333 H=2333:00 +CPU= 16.758
  23:49:38 STEP 2334 H=2334:00 +CPU= 26.886
  23:49:54 STEP 2335 H=2335:00 +CPU= 16.897
  23:50:12 STEP 2336 H=2336:00 +CPU= 17.158
  23:50:38 STEP 2337 H=2337:00 +CPU= 26.707
  23:50:56 STEP 2338 H=2338:00 +CPU= 17.265
  23:51:12 STEP 2339 H=2339:00 +CPU= 16.729
  23:51:42 STEP 2340 H=2340:00 +CPU= 29.952
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002340
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002340
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002340
  23:51:59 STEP 2341 H=2341:00 +CPU= 16.785
  23:52:16 STEP 2342 H=2342:00 +CPU= 17.231
  23:52:43 STEP 2343 H=2343:00 +CPU= 26.444
  23:53:00 STEP 2344 H=2344:00 +CPU= 17.183
  23:53:16 STEP 2345 H=2345:00 +CPU= 16.483
  23:53:43 STEP 2346 H=2346:00 +CPU= 26.645
  23:54:00 STEP 2347 H=2347:00 +CPU= 16.683
  23:54:16 STEP 2348 H=2348:00 +CPU= 16.587
  23:55:01 STEP 2349 H=2349:00 +CPU= 28.366
  23:55:18 STEP 2350 H=2350:00 +CPU= 16.581
  23:55:37 STEP 2351 H=2351:00 +CPU= 18.523
  23:56:23 STEP 2352 H=2352:00 +CPU= 31.048
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002352
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002352
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002328
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002340
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002340
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_97.zip
Uploading the intermediate file: upload_file_97.zip
  23:56:39 STEP 2353 H=2353:00 +CPU= 16.489
  23:56:57 STEP 2354 H=2354:00 +CPU= 17.236
  23:57:24 STEP 2355 H=2355:00 +CPU= 26.867
  23:57:40 STEP 2356 H=2356:00 +CPU= 16.845
  23:57:58 STEP 2357 H=2357:00 +CPU= 17.327
  23:58:26 STEP 2358 H=2358:00 +CPU= 27.878
  23:58:42 STEP 2359 H=2359:00 +CPU= 16.759
  23:59:00 STEP 2360 H=2360:00 +CPU= 17.560
  23:59:26 STEP 2361 H=2361:00 +CPU= 26.353
  23:59:45 STEP 2362 H=2362:00 +CPU= 18.340
  00:00:02 STEP 2363 H=2363:00 +CPU= 17.393
  00:00:32 STEP 2364 H=2364:00 +CPU= 29.530
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002364
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002364
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002364
  00:00:48 STEP 2365 H=2365:00 +CPU= 16.385
  00:01:06 STEP 2366 H=2366:00 +CPU= 18.235
  00:01:33 STEP 2367 H=2367:00 +CPU= 26.874
  00:01:50 STEP 2368 H=2368:00 +CPU= 17.205
  00:02:07 STEP 2369 H=2369:00 +CPU= 16.458
  00:02:34 STEP 2370 H=2370:00 +CPU= 27.318
  00:02:51 STEP 2371 H=2371:00 +CPU= 16.656
  00:03:08 STEP 2372 H=2372:00 +CPU= 16.933
  00:03:34 STEP 2373 H=2373:00 +CPU= 26.069
  00:03:51 STEP 2374 H=2374:00 +CPU= 17.034
  00:04:08 STEP 2375 H=2375:00 +CPU= 16.751
  00:04:38 STEP 2376 H=2376:00 +CPU= 30.037
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002376
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002376
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002352
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002364
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002364
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_98.zip
Uploading the intermediate file: upload_file_98.zip
  00:04:54 STEP 2377 H=2377:00 +CPU= 16.503
  00:05:11 STEP 2378 H=2378:00 +CPU= 16.830
  00:05:38 STEP 2379 H=2379:00 +CPU= 26.911
  00:05:55 STEP 2380 H=2380:00 +CPU= 17.308
  00:06:12 STEP 2381 H=2381:00 +CPU= 16.160
  00:06:40 STEP 2382 H=2382:00 +CPU= 27.903
  00:06:56 STEP 2383 H=2383:00 +CPU= 16.602
  00:07:13 STEP 2384 H=2384:00 +CPU= 16.466
  00:07:39 STEP 2385 H=2385:00 +CPU= 26.705
  00:07:56 STEP 2386 H=2386:00 +CPU= 17.051
  00:08:12 STEP 2387 H=2387:00 +CPU= 16.005
  00:08:43 STEP 2388 H=2388:00 +CPU= 30.933
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002388
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002388
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002388
  00:09:00 STEP 2389 H=2389:00 +CPU= 16.527
  00:09:17 STEP 2390 H=2390:00 +CPU= 16.924
  00:09:43 STEP 2391 H=2391:00 +CPU= 26.215
  00:10:01 STEP 2392 H=2392:00 +CPU= 17.656
  00:10:17 STEP 2393 H=2393:00 +CPU= 16.660
  00:10:44 STEP 2394 H=2394:00 +CPU= 26.447
  00:11:00 STEP 2395 H=2395:00 +CPU= 16.231
  00:11:17 STEP 2396 H=2396:00 +CPU= 16.837
  00:11:43 STEP 2397 H=2397:00 +CPU= 26.383
  00:12:00 STEP 2398 H=2398:00 +CPU= 16.837
  00:12:17 STEP 2399 H=2399:00 +CPU= 16.504
  00:12:47 STEP 2400 H=2400:00 +CPU= 30.719
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002400
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002400
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002376
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002388
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002388
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_99.zip
Uploading the intermediate file: upload_file_99.zip
  00:13:04 STEP 2401 H=2401:00 +CPU= 16.720
  00:13:21 STEP 2402 H=2402:00 +CPU= 16.677
  00:13:47 STEP 2403 H=2403:00 +CPU= 26.499
  00:14:04 STEP 2404 H=2404:00 +CPU= 16.927
  00:14:20 STEP 2405 H=2405:00 +CPU= 16.100
  00:14:48 STEP 2406 H=2406:00 +CPU= 27.729
  00:15:04 STEP 2407 H=2407:00 +CPU= 16.270
  00:15:21 STEP 2408 H=2408:00 +CPU= 16.539
  00:15:48 STEP 2409 H=2409:00 +CPU= 26.770
  00:16:05 STEP 2410 H=2410:00 +CPU= 17.000
  00:16:21 STEP 2411 H=2411:00 +CPU= 16.553
  00:16:50 STEP 2412 H=2412:00 +CPU= 29.027
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002412
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002412
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002412
  00:17:07 STEP 2413 H=2413:00 +CPU= 16.633
  00:17:24 STEP 2414 H=2414:00 +CPU= 17.213
  00:17:50 STEP 2415 H=2415:00 +CPU= 25.578
  00:18:07 STEP 2416 H=2416:00 +CPU= 16.723
  00:18:23 STEP 2417 H=2417:00 +CPU= 16.730
  00:18:50 STEP 2418 H=2418:00 +CPU= 26.625
  00:19:06 STEP 2419 H=2419:00 +CPU= 15.839
  00:19:24 STEP 2420 H=2420:00 +CPU= 17.963
  00:19:50 STEP 2421 H=2421:00 +CPU= 26.605
  00:20:07 STEP 2422 H=2422:00 +CPU= 16.731
  00:20:24 STEP 2423 H=2423:00 +CPU= 16.479
  00:20:59 STEP 2424 H=2424:00 +CPU= 30.262
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002424
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002424
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002400
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002412
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002412
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_100.zip
Uploading the intermediate file: upload_file_100.zip
  00:21:16 STEP 2425 H=2425:00 +CPU= 16.967
  00:21:33 STEP 2426 H=2426:00 +CPU= 16.949
  00:22:00 STEP 2427 H=2427:00 +CPU= 26.484
  00:22:17 STEP 2428 H=2428:00 +CPU= 17.357
  00:22:35 STEP 2429 H=2429:00 +CPU= 17.573
  00:23:03 STEP 2430 H=2430:00 +CPU= 28.481
  00:23:20 STEP 2431 H=2431:00 +CPU= 16.489
  00:23:37 STEP 2432 H=2432:00 +CPU= 17.219
  00:24:04 STEP 2433 H=2433:00 +CPU= 27.207
  00:24:22 STEP 2434 H=2434:00 +CPU= 17.571
  00:24:39 STEP 2435 H=2435:00 +CPU= 17.431
  00:25:11 STEP 2436 H=2436:00 +CPU= 31.517
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002436
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002436
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002436
  00:25:29 STEP 2437 H=2437:00 +CPU= 17.914
  00:25:47 STEP 2438 H=2438:00 +CPU= 18.342
  00:26:14 STEP 2439 H=2439:00 +CPU= 27.058
  00:26:32 STEP 2440 H=2440:00 +CPU= 18.230
  00:26:50 STEP 2441 H=2441:00 +CPU= 17.461
  00:27:17 STEP 2442 H=2442:00 +CPU= 26.888
  00:27:34 STEP 2443 H=2443:00 +CPU= 17.464
  00:27:51 STEP 2444 H=2444:00 +CPU= 17.286
  00:28:19 STEP 2445 H=2445:00 +CPU= 27.279
  00:28:37 STEP 2446 H=2446:00 +CPU= 18.367
  00:28:55 STEP 2447 H=2447:00 +CPU= 18.157
  00:29:27 STEP 2448 H=2448:00 +CPU= 31.560
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002448
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002448
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002424
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002436
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002436
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_101.zip
Uploading the intermediate file: upload_file_101.zip
  00:29:44 STEP 2449 H=2449:00 +CPU= 16.988
  00:30:02 STEP 2450 H=2450:00 +CPU= 18.457
  00:30:29 STEP 2451 H=2451:00 +CPU= 26.424
  00:30:46 STEP 2452 H=2452:00 +CPU= 16.863
  00:31:02 STEP 2453 H=2453:00 +CPU= 16.829
  00:31:31 STEP 2454 H=2454:00 +CPU= 28.589
  00:31:47 STEP 2455 H=2455:00 +CPU= 16.315
  00:32:04 STEP 2456 H=2456:00 +CPU= 17.021
  00:32:30 STEP 2457 H=2457:00 +CPU= 25.627
  00:32:47 STEP 2458 H=2458:00 +CPU= 16.869
  00:33:03 STEP 2459 H=2459:00 +CPU= 16.468
  00:33:32 STEP 2460 H=2460:00 +CPU= 28.995
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002460
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002460
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002460
  00:33:49 STEP 2461 H=2461:00 +CPU= 16.537
  00:34:06 STEP 2462 H=2462:00 +CPU= 16.939
  00:34:33 STEP 2463 H=2463:00 +CPU= 27.271
  00:34:51 STEP 2464 H=2464:00 +CPU= 17.639
  00:35:09 STEP 2465 H=2465:00 +CPU= 17.874
  00:35:35 STEP 2466 H=2466:00 +CPU= 26.516
  00:35:52 STEP 2467 H=2467:00 +CPU= 17.086
  00:36:10 STEP 2468 H=2468:00 +CPU= 17.652
  00:36:36 STEP 2469 H=2469:00 +CPU= 26.308
  00:36:53 STEP 2470 H=2470:00 +CPU= 16.654
  00:37:11 STEP 2471 H=2471:00 +CPU= 17.847
  00:37:43 STEP 2472 H=2472:00 +CPU= 31.249
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002472
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002472
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002448
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002460
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002460
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_102.zip
Uploading the intermediate file: upload_file_102.zip
  00:37:59 STEP 2473 H=2473:00 +CPU= 16.240
  00:38:17 STEP 2474 H=2474:00 +CPU= 17.798
  00:38:43 STEP 2475 H=2475:00 +CPU= 25.999
  00:39:00 STEP 2476 H=2476:00 +CPU= 17.345
  00:39:17 STEP 2477 H=2477:00 +CPU= 16.561
  00:39:43 STEP 2478 H=2478:00 +CPU= 26.660
  00:39:59 STEP 2479 H=2479:00 +CPU= 16.083
  00:40:16 STEP 2480 H=2480:00 +CPU= 17.085
  00:40:42 STEP 2481 H=2481:00 +CPU= 25.756
  00:40:59 STEP 2482 H=2482:00 +CPU= 16.682
  00:41:16 STEP 2483 H=2483:00 +CPU= 16.729
  00:41:44 STEP 2484 H=2484:00 +CPU= 28.834
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002484
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002484
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002484
  00:42:01 STEP 2485 H=2485:00 +CPU= 16.566
  00:42:18 STEP 2486 H=2486:00 +CPU= 17.311
  00:42:44 STEP 2487 H=2487:00 +CPU= 26.178
  00:43:01 STEP 2488 H=2488:00 +CPU= 17.000
  00:43:18 STEP 2489 H=2489:00 +CPU= 16.940
  00:43:45 STEP 2490 H=2490:00 +CPU= 26.895
  00:44:03 STEP 2491 H=2491:00 +CPU= 17.262
  00:44:20 STEP 2492 H=2492:00 +CPU= 17.548
  00:44:46 STEP 2493 H=2493:00 +CPU= 25.657
  00:45:03 STEP 2494 H=2494:00 +CPU= 17.130
  00:45:20 STEP 2495 H=2495:00 +CPU= 16.984
  00:45:52 STEP 2496 H=2496:00 +CPU= 30.809
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002496
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002496
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002472
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002484
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002484
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_103.zip
Uploading the intermediate file: upload_file_103.zip
  00:46:08 STEP 2497 H=2497:00 +CPU= 16.113
  00:46:25 STEP 2498 H=2498:00 +CPU= 17.393
  00:46:51 STEP 2499 H=2499:00 +CPU= 26.252
  00:47:09 STEP 2500 H=2500:00 +CPU= 17.436
  00:47:26 STEP 2501 H=2501:00 +CPU= 16.921
  00:47:52 STEP 2502 H=2502:00 +CPU= 26.622
  00:48:09 STEP 2503 H=2503:00 +CPU= 16.631
  00:48:26 STEP 2504 H=2504:00 +CPU= 16.665
  00:48:52 STEP 2505 H=2505:00 +CPU= 26.112
  00:49:09 STEP 2506 H=2506:00 +CPU= 16.731
  00:49:25 STEP 2507 H=2507:00 +CPU= 16.291
  00:49:54 STEP 2508 H=2508:00 +CPU= 28.802
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002508
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002508
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002508
  00:50:10 STEP 2509 H=2509:00 +CPU= 16.453
  00:50:27 STEP 2510 H=2510:00 +CPU= 17.316
  00:50:53 STEP 2511 H=2511:00 +CPU= 25.587
  00:51:10 STEP 2512 H=2512:00 +CPU= 16.899
  00:51:26 STEP 2513 H=2513:00 +CPU= 16.227
  00:51:53 STEP 2514 H=2514:00 +CPU= 26.720
  00:52:10 STEP 2515 H=2515:00 +CPU= 16.899
  00:52:27 STEP 2516 H=2516:00 +CPU= 17.152
  00:52:54 STEP 2517 H=2517:00 +CPU= 26.513
  00:53:11 STEP 2518 H=2518:00 +CPU= 17.600
  00:53:28 STEP 2519 H=2519:00 +CPU= 16.741
  00:53:58 STEP 2520 H=2520:00 +CPU= 29.662
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002520
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002520
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002496
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002508
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002508
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_104.zip
Uploading the intermediate file: upload_file_104.zip
  00:54:15 STEP 2521 H=2521:00 +CPU= 16.875
  00:54:32 STEP 2522 H=2522:00 +CPU= 16.887
  00:54:58 STEP 2523 H=2523:00 +CPU= 25.970
  00:55:16 STEP 2524 H=2524:00 +CPU= 17.979
  00:55:33 STEP 2525 H=2525:00 +CPU= 17.150
  00:56:00 STEP 2526 H=2526:00 +CPU= 27.138
  00:56:16 STEP 2527 H=2527:00 +CPU= 16.546
  00:56:33 STEP 2528 H=2528:00 +CPU= 16.434
  00:56:59 STEP 2529 H=2529:00 +CPU= 26.413
  00:57:16 STEP 2530 H=2530:00 +CPU= 16.582
  00:57:32 STEP 2531 H=2531:00 +CPU= 15.786
  00:58:02 STEP 2532 H=2532:00 +CPU= 30.150
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002532
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002532
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002532
  00:58:18 STEP 2533 H=2533:00 +CPU= 16.107
  00:58:35 STEP 2534 H=2534:00 +CPU= 17.250
  00:59:02 STEP 2535 H=2535:00 +CPU= 26.766
  00:59:19 STEP 2536 H=2536:00 +CPU= 16.688
  00:59:34 STEP 2537 H=2537:00 +CPU= 15.723
  01:00:02 STEP 2538 H=2538:00 +CPU= 27.962
  01:00:19 STEP 2539 H=2539:00 +CPU= 16.421
  01:00:36 STEP 2540 H=2540:00 +CPU= 16.972
  01:01:04 STEP 2541 H=2541:00 +CPU= 28.156
  01:01:22 STEP 2542 H=2542:00 +CPU= 17.850
  01:01:39 STEP 2543 H=2543:00 +CPU= 17.285
  01:02:13 STEP 2544 H=2544:00 +CPU= 32.778
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002544
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002544
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002520
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002532
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002532
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_105.zip
Uploading the intermediate file: upload_file_105.zip
  01:02:30 STEP 2545 H=2545:00 +CPU= 16.672
Uploading trickle at timestep: 9158400
  01:02:48 STEP 2546 H=2546:00 +CPU= 18.576
  01:03:15 STEP 2547 H=2547:00 +CPU= 26.688
  01:03:33 STEP 2548 H=2548:00 +CPU= 18.268
  01:03:51 STEP 2549 H=2549:00 +CPU= 17.591
  01:04:19 STEP 2550 H=2550:00 +CPU= 27.702
  01:04:35 STEP 2551 H=2551:00 +CPU= 16.876
  01:04:55 STEP 2552 H=2552:00 +CPU= 19.412
  01:05:22 STEP 2553 H=2553:00 +CPU= 26.897
  01:05:41 STEP 2554 H=2554:00 +CPU= 19.273
  01:05:59 STEP 2555 H=2555:00 +CPU= 17.963
  01:06:34 STEP 2556 H=2556:00 +CPU= 34.625
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002556
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002556
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002556
  01:06:50 STEP 2557 H=2557:00 +CPU= 16.764
  01:07:09 STEP 2558 H=2558:00 +CPU= 18.300
  01:07:39 STEP 2559 H=2559:00 +CPU= 30.195
  01:07:57 STEP 2560 H=2560:00 +CPU= 18.370
  01:08:15 STEP 2561 H=2561:00 +CPU= 17.295
  01:08:42 STEP 2562 H=2562:00 +CPU= 27.224
  01:08:59 STEP 2563 H=2563:00 +CPU= 16.674
  01:09:16 STEP 2564 H=2564:00 +CPU= 17.704
  01:09:44 STEP 2565 H=2565:00 +CPU= 27.057
  01:10:02 STEP 2566 H=2566:00 +CPU= 18.126
  01:10:19 STEP 2567 H=2567:00 +CPU= 17.501
  01:10:51 STEP 2568 H=2568:00 +CPU= 31.197
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002568
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002568
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002544
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002556
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002556
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_106.zip
Uploading the intermediate file: upload_file_106.zip
  01:11:10 STEP 2569 H=2569:00 +CPU= 18.337
  01:11:28 STEP 2570 H=2570:00 +CPU= 18.541
  01:11:56 STEP 2571 H=2571:00 +CPU= 27.872
  01:12:15 STEP 2572 H=2572:00 +CPU= 18.883
  01:12:34 STEP 2573 H=2573:00 +CPU= 19.305
  01:13:03 STEP 2574 H=2574:00 +CPU= 28.689
  01:13:20 STEP 2575 H=2575:00 +CPU= 17.117
  01:13:39 STEP 2576 H=2576:00 +CPU= 19.102
  01:14:08 STEP 2577 H=2577:00 +CPU= 29.195
  01:14:26 STEP 2578 H=2578:00 +CPU= 17.547
  01:14:43 STEP 2579 H=2579:00 +CPU= 16.797
  01:15:13 STEP 2580 H=2580:00 +CPU= 30.452
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002580
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002580
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002580
  01:15:31 STEP 2581 H=2581:00 +CPU= 17.369
  01:15:48 STEP 2582 H=2582:00 +CPU= 17.362
  01:16:16 STEP 2583 H=2583:00 +CPU= 28.398
  01:16:34 STEP 2584 H=2584:00 +CPU= 17.487
  01:16:51 STEP 2585 H=2585:00 +CPU= 16.765
  01:17:19 STEP 2586 H=2586:00 +CPU= 28.400
  01:17:36 STEP 2587 H=2587:00 +CPU= 16.884
  01:17:56 STEP 2588 H=2588:00 +CPU= 19.643
  01:18:24 STEP 2589 H=2589:00 +CPU= 28.036
  01:18:42 STEP 2590 H=2590:00 +CPU= 17.800
  01:19:00 STEP 2591 H=2591:00 +CPU= 18.136
  01:19:33 STEP 2592 H=2592:00 +CPU= 31.162
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002592
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002592
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002568
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002580
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002580
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_107.zip
Uploading the intermediate file: upload_file_107.zip
  01:19:50 STEP 2593 H=2593:00 +CPU= 17.090
  01:20:08 STEP 2594 H=2594:00 +CPU= 18.113
  01:20:34 STEP 2595 H=2595:00 +CPU= 26.436
  01:20:51 STEP 2596 H=2596:00 +CPU= 16.544
  01:21:07 STEP 2597 H=2597:00 +CPU= 16.321
  01:21:34 STEP 2598 H=2598:00 +CPU= 26.989
  01:21:51 STEP 2599 H=2599:00 +CPU= 17.050
  01:22:10 STEP 2600 H=2600:00 +CPU= 18.056
  01:22:36 STEP 2601 H=2601:00 +CPU= 26.087
  01:22:54 STEP 2602 H=2602:00 +CPU= 17.939
  01:23:10 STEP 2603 H=2603:00 +CPU= 16.906
  01:23:39 STEP 2604 H=2604:00 +CPU= 29.001
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002604
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002604
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002604
  01:23:56 STEP 2605 H=2605:00 +CPU= 16.720
  01:24:13 STEP 2606 H=2606:00 +CPU= 17.140
  01:24:40 STEP 2607 H=2607:00 +CPU= 26.266
  01:24:57 STEP 2608 H=2608:00 +CPU= 17.270
  01:25:13 STEP 2609 H=2609:00 +CPU= 16.515
  01:25:41 STEP 2610 H=2610:00 +CPU= 27.102
  01:25:56 STEP 2611 H=2611:00 +CPU= 15.711
  01:26:12 STEP 2612 H=2612:00 +CPU= 15.937
  01:26:37 STEP 2613 H=2613:00 +CPU= 24.385
  01:26:54 STEP 2614 H=2614:00 +CPU= 17.053
  01:27:09 STEP 2615 H=2615:00 +CPU= 15.711
  01:27:38 STEP 2616 H=2616:00 +CPU= 28.253
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002616
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002616
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002592
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002604
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002604
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_108.zip
Uploading the intermediate file: upload_file_108.zip
  01:27:54 STEP 2617 H=2617:00 +CPU= 15.858
  01:28:10 STEP 2618 H=2618:00 +CPU= 15.791
  01:28:34 STEP 2619 H=2619:00 +CPU= 24.153
  01:28:50 STEP 2620 H=2620:00 +CPU= 15.916
  01:29:05 STEP 2621 H=2621:00 +CPU= 14.912
  01:29:31 STEP 2622 H=2622:00 +CPU= 26.486
  01:29:46 STEP 2623 H=2623:00 +CPU= 15.081
  01:30:02 STEP 2624 H=2624:00 +CPU= 15.552
  01:30:27 STEP 2625 H=2625:00 +CPU= 24.891
  01:30:43 STEP 2626 H=2626:00 +CPU= 16.094
  01:30:59 STEP 2627 H=2627:00 +CPU= 16.217
  01:31:27 STEP 2628 H=2628:00 +CPU= 27.802
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002628
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002628
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002628
  01:31:43 STEP 2629 H=2629:00 +CPU= 15.723
  01:31:59 STEP 2630 H=2630:00 +CPU= 16.557
  01:32:24 STEP 2631 H=2631:00 +CPU= 24.937
  01:32:41 STEP 2632 H=2632:00 +CPU= 16.820
  01:32:58 STEP 2633 H=2633:00 +CPU= 16.014
  01:33:23 STEP 2634 H=2634:00 +CPU= 25.262
  01:33:38 STEP 2635 H=2635:00 +CPU= 14.998
  01:33:53 STEP 2636 H=2636:00 +CPU= 14.973
  01:34:18 STEP 2637 H=2637:00 +CPU= 24.581
  01:34:34 STEP 2638 H=2638:00 +CPU= 16.168
  01:34:50 STEP 2639 H=2639:00 +CPU= 16.170
  01:35:21 STEP 2640 H=2640:00 +CPU= 29.569
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002640
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002640
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002616
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002628
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002628
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_109.zip
Uploading the intermediate file: upload_file_109.zip
  01:35:36 STEP 2641 H=2641:00 +CPU= 15.672
  01:35:52 STEP 2642 H=2642:00 +CPU= 15.381
  01:36:15 STEP 2643 H=2643:00 +CPU= 23.480
  01:36:31 STEP 2644 H=2644:00 +CPU= 15.457
  01:36:47 STEP 2645 H=2645:00 +CPU= 15.946
  01:37:12 STEP 2646 H=2646:00 +CPU= 25.511
  01:37:28 STEP 2647 H=2647:00 +CPU= 15.859
  01:37:45 STEP 2648 H=2648:00 +CPU= 16.533
  01:38:10 STEP 2649 H=2649:00 +CPU= 25.191
  01:38:27 STEP 2650 H=2650:00 +CPU= 16.825
  01:38:43 STEP 2651 H=2651:00 +CPU= 16.306
  01:39:11 STEP 2652 H=2652:00 +CPU= 28.295
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002652
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002652
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002652
  01:39:27 STEP 2653 H=2653:00 +CPU= 16.004
  01:39:44 STEP 2654 H=2654:00 +CPU= 16.788
  01:40:10 STEP 2655 H=2655:00 +CPU= 25.689
  01:40:26 STEP 2656 H=2656:00 +CPU= 16.469
  01:40:43 STEP 2657 H=2657:00 +CPU= 16.620
  01:41:09 STEP 2658 H=2658:00 +CPU= 25.785
  01:41:25 STEP 2659 H=2659:00 +CPU= 15.998
  01:41:41 STEP 2660 H=2660:00 +CPU= 16.928
  01:42:07 STEP 2661 H=2661:00 +CPU= 25.572
  01:42:24 STEP 2662 H=2662:00 +CPU= 16.772
  01:42:40 STEP 2663 H=2663:00 +CPU= 16.072
  01:43:17 STEP 2664 H=2664:00 +CPU= 29.678
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002664
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002664
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002640
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002652
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002652
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_110.zip
Uploading the intermediate file: upload_file_110.zip
  01:43:33 STEP 2665 H=2665:00 +CPU= 15.785
  01:43:49 STEP 2666 H=2666:00 +CPU= 16.381
  01:44:15 STEP 2667 H=2667:00 +CPU= 25.767
  01:44:31 STEP 2668 H=2668:00 +CPU= 16.684
  01:44:48 STEP 2669 H=2669:00 +CPU= 16.491
  01:45:14 STEP 2670 H=2670:00 +CPU= 26.307
  01:45:30 STEP 2671 H=2671:00 +CPU= 16.055
  01:45:47 STEP 2672 H=2672:00 +CPU= 16.652
  01:46:13 STEP 2673 H=2673:00 +CPU= 25.858
  01:46:30 STEP 2674 H=2674:00 +CPU= 17.203
  01:46:46 STEP 2675 H=2675:00 +CPU= 16.100
  01:47:15 STEP 2676 H=2676:00 +CPU= 29.084
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002676
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002676
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002676
  01:47:32 STEP 2677 H=2677:00 +CPU= 16.633
  01:47:49 STEP 2678 H=2678:00 +CPU= 17.054
  01:48:15 STEP 2679 H=2679:00 +CPU= 25.977
  01:48:32 STEP 2680 H=2680:00 +CPU= 16.651
  01:48:48 STEP 2681 H=2681:00 +CPU= 16.338
  01:49:15 STEP 2682 H=2682:00 +CPU= 26.780
  01:49:31 STEP 2683 H=2683:00 +CPU= 16.437
  01:49:49 STEP 2684 H=2684:00 +CPU= 17.525
  01:50:15 STEP 2685 H=2685:00 +CPU= 26.442
  01:50:32 STEP 2686 H=2686:00 +CPU= 17.239
  01:50:49 STEP 2687 H=2687:00 +CPU= 17.064
  01:51:20 STEP 2688 H=2688:00 +CPU= 30.208
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002688
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002688
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002664
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002676
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002676
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_111.zip
Uploading the intermediate file: upload_file_111.zip
  01:51:36 STEP 2689 H=2689:00 +CPU= 16.374
  01:51:53 STEP 2690 H=2690:00 +CPU= 17.071
  01:52:20 STEP 2691 H=2691:00 +CPU= 26.547
  01:52:37 STEP 2692 H=2692:00 +CPU= 16.992
  01:52:53 STEP 2693 H=2693:00 +CPU= 16.599
  01:53:20 STEP 2694 H=2694:00 +CPU= 26.398
  01:53:36 STEP 2695 H=2695:00 +CPU= 16.212
  01:53:53 STEP 2696 H=2696:00 +CPU= 17.018
  01:54:19 STEP 2697 H=2697:00 +CPU= 25.977
  01:54:36 STEP 2698 H=2698:00 +CPU= 16.924
  01:54:52 STEP 2699 H=2699:00 +CPU= 15.649
  01:55:21 STEP 2700 H=2700:00 +CPU= 29.186
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002700
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002700
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002700
  01:55:38 STEP 2701 H=2701:00 +CPU= 17.269
  01:55:55 STEP 2702 H=2702:00 +CPU= 17.426
  01:56:22 STEP 2703 H=2703:00 +CPU= 26.583
  01:56:39 STEP 2704 H=2704:00 +CPU= 17.286
  01:56:56 STEP 2705 H=2705:00 +CPU= 16.337
  01:57:22 STEP 2706 H=2706:00 +CPU= 26.336
  01:57:40 STEP 2707 H=2707:00 +CPU= 17.571
  01:57:56 STEP 2708 H=2708:00 +CPU= 16.545
  01:58:22 STEP 2709 H=2709:00 +CPU= 25.742
  01:58:40 STEP 2710 H=2710:00 +CPU= 18.372
  01:58:57 STEP 2711 H=2711:00 +CPU= 16.636
  01:59:28 STEP 2712 H=2712:00 +CPU= 31.053
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002712
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002712
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002688
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002700
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002700
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_112.zip
Uploading the intermediate file: upload_file_112.zip
  01:59:45 STEP 2713 H=2713:00 +CPU= 16.881
  02:00:02 STEP 2714 H=2714:00 +CPU= 16.784
  02:00:28 STEP 2715 H=2715:00 +CPU= 25.987
  02:00:45 STEP 2716 H=2716:00 +CPU= 17.638
  02:01:02 STEP 2717 H=2717:00 +CPU= 16.338
  02:01:29 STEP 2718 H=2718:00 +CPU= 27.011
  02:01:45 STEP 2719 H=2719:00 +CPU= 16.737
  02:02:02 STEP 2720 H=2720:00 +CPU= 16.710
  02:02:29 STEP 2721 H=2721:00 +CPU= 26.563
  02:02:46 STEP 2722 H=2722:00 +CPU= 17.075
  02:03:02 STEP 2723 H=2723:00 +CPU= 16.065
  02:03:31 STEP 2724 H=2724:00 +CPU= 28.636
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002724
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002724
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002724
  02:03:47 STEP 2725 H=2725:00 +CPU= 16.247
  02:04:04 STEP 2726 H=2726:00 +CPU= 17.143
  02:04:30 STEP 2727 H=2727:00 +CPU= 26.303
  02:04:47 STEP 2728 H=2728:00 +CPU= 16.545
  02:05:04 STEP 2729 H=2729:00 +CPU= 16.851
  02:05:30 STEP 2730 H=2730:00 +CPU= 26.716
  02:05:46 STEP 2731 H=2731:00 +CPU= 15.985
  02:06:04 STEP 2732 H=2732:00 +CPU= 17.658
  02:06:31 STEP 2733 H=2733:00 +CPU= 26.746
  02:06:49 STEP 2734 H=2734:00 +CPU= 17.769
  02:07:06 STEP 2735 H=2735:00 +CPU= 17.179
  02:07:36 STEP 2736 H=2736:00 +CPU= 30.735
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002736
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002736
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002712
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002724
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002724
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_113.zip
Uploading the intermediate file: upload_file_113.zip
  02:07:52 STEP 2737 H=2737:00 +CPU= 15.949
  02:08:09 STEP 2738 H=2738:00 +CPU= 16.528
  02:08:35 STEP 2739 H=2739:00 +CPU= 26.045
  02:08:51 STEP 2740 H=2740:00 +CPU= 16.451
  02:09:07 STEP 2741 H=2741:00 +CPU= 15.758
  02:09:34 STEP 2742 H=2742:00 +CPU= 26.918
  02:09:50 STEP 2743 H=2743:00 +CPU= 16.091
  02:10:07 STEP 2744 H=2744:00 +CPU= 17.071
  02:10:33 STEP 2745 H=2745:00 +CPU= 25.330
  02:10:49 STEP 2746 H=2746:00 +CPU= 16.754
  02:11:06 STEP 2747 H=2747:00 +CPU= 16.257
  02:11:34 STEP 2748 H=2748:00 +CPU= 28.768
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002748
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002748
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002748
  02:11:50 STEP 2749 H=2749:00 +CPU= 16.086
  02:12:07 STEP 2750 H=2750:00 +CPU= 16.908
  02:12:33 STEP 2751 H=2751:00 +CPU= 25.442
  02:12:50 STEP 2752 H=2752:00 +CPU= 16.779
  02:13:06 STEP 2753 H=2753:00 +CPU= 16.449
  02:13:33 STEP 2754 H=2754:00 +CPU= 26.475
  02:13:49 STEP 2755 H=2755:00 +CPU= 16.621
  02:14:06 STEP 2756 H=2756:00 +CPU= 16.693
  02:14:31 STEP 2757 H=2757:00 +CPU= 25.304
  02:14:48 STEP 2758 H=2758:00 +CPU= 16.866
  02:15:05 STEP 2759 H=2759:00 +CPU= 17.091
  02:15:36 STEP 2760 H=2760:00 +CPU= 30.117
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002760
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002760
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002736
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002748
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002748
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_114.zip
Uploading the intermediate file: upload_file_114.zip
  02:15:53 STEP 2761 H=2761:00 +CPU= 16.663
  02:16:09 STEP 2762 H=2762:00 +CPU= 16.669
  02:16:35 STEP 2763 H=2763:00 +CPU= 25.416
  02:16:52 STEP 2764 H=2764:00 +CPU= 16.941
  02:17:08 STEP 2765 H=2765:00 +CPU= 15.897
  02:17:34 STEP 2766 H=2766:00 +CPU= 26.292
  02:17:50 STEP 2767 H=2767:00 +CPU= 16.306
  02:18:07 STEP 2768 H=2768:00 +CPU= 16.610
  02:18:33 STEP 2769 H=2769:00 +CPU= 26.132
  02:18:50 STEP 2770 H=2770:00 +CPU= 16.958
  02:19:07 STEP 2771 H=2771:00 +CPU= 16.662
  02:19:36 STEP 2772 H=2772:00 +CPU= 29.055
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002772
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002772
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002772
  02:19:52 STEP 2773 H=2773:00 +CPU= 16.841
  02:20:09 STEP 2774 H=2774:00 +CPU= 16.881
  02:20:35 STEP 2775 H=2775:00 +CPU= 25.903
  02:20:52 STEP 2776 H=2776:00 +CPU= 16.459
  02:21:08 STEP 2777 H=2777:00 +CPU= 15.998
  02:21:35 STEP 2778 H=2778:00 +CPU= 27.496
  02:21:52 STEP 2779 H=2779:00 +CPU= 16.416
  02:22:09 STEP 2780 H=2780:00 +CPU= 16.900
  02:22:34 STEP 2781 H=2781:00 +CPU= 25.629
  02:22:51 STEP 2782 H=2782:00 +CPU= 16.773
  02:23:07 STEP 2783 H=2783:00 +CPU= 16.062
  02:23:43 STEP 2784 H=2784:00 +CPU= 30.058
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002784
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002784
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002760
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002772
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002772
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_115.zip
Uploading the intermediate file: upload_file_115.zip
  02:23:59 STEP 2785 H=2785:00 +CPU= 16.005
Uploading trickle at timestep: 10022400
  02:24:17 STEP 2786 H=2786:00 +CPU= 17.414
  02:24:43 STEP 2787 H=2787:00 +CPU= 26.104
  02:25:00 STEP 2788 H=2788:00 +CPU= 16.781
  02:25:17 STEP 2789 H=2789:00 +CPU= 17.020
  02:25:43 STEP 2790 H=2790:00 +CPU= 26.773
  02:25:59 STEP 2791 H=2791:00 +CPU= 15.720
  02:26:16 STEP 2792 H=2792:00 +CPU= 16.668
  02:26:41 STEP 2793 H=2793:00 +CPU= 24.774
  02:26:57 STEP 2794 H=2794:00 +CPU= 16.746
  02:27:14 STEP 2795 H=2795:00 +CPU= 16.516
  02:27:42 STEP 2796 H=2796:00 +CPU= 27.776
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002796
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002796
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002796
  02:27:58 STEP 2797 H=2797:00 +CPU= 16.557
  02:28:15 STEP 2798 H=2798:00 +CPU= 17.329
  02:28:41 STEP 2799 H=2799:00 +CPU= 25.470
  02:28:58 STEP 2800 H=2800:00 +CPU= 17.278
  02:29:15 STEP 2801 H=2801:00 +CPU= 16.624
  02:29:41 STEP 2802 H=2802:00 +CPU= 26.482
  02:29:58 STEP 2803 H=2803:00 +CPU= 16.463
  02:30:14 STEP 2804 H=2804:00 +CPU= 16.679
  02:30:40 STEP 2805 H=2805:00 +CPU= 25.341
  02:30:56 STEP 2806 H=2806:00 +CPU= 16.481
  02:31:13 STEP 2807 H=2807:00 +CPU= 16.636
  02:31:44 STEP 2808 H=2808:00 +CPU= 30.188
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002808
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002808
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002784
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002796
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002796
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_116.zip
Uploading the intermediate file: upload_file_116.zip
  02:32:01 STEP 2809 H=2809:00 +CPU= 16.675
  02:32:18 STEP 2810 H=2810:00 +CPU= 16.729
  02:32:44 STEP 2811 H=2811:00 +CPU= 25.975
  02:33:01 STEP 2812 H=2812:00 +CPU= 17.606
  02:33:17 STEP 2813 H=2813:00 +CPU= 16.101
  02:33:43 STEP 2814 H=2814:00 +CPU= 25.868
  02:34:00 STEP 2815 H=2815:00 +CPU= 16.354
  02:34:17 STEP 2816 H=2816:00 +CPU= 17.166
  02:34:43 STEP 2817 H=2817:00 +CPU= 25.992
  02:35:00 STEP 2818 H=2818:00 +CPU= 17.547
  02:35:19 STEP 2819 H=2819:00 +CPU= 18.162
  02:35:49 STEP 2820 H=2820:00 +CPU= 30.059
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002820
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002820
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002820
  02:36:07 STEP 2821 H=2821:00 +CPU= 17.404
  02:36:24 STEP 2822 H=2822:00 +CPU= 17.539
  02:36:52 STEP 2823 H=2823:00 +CPU= 27.310
  02:37:10 STEP 2824 H=2824:00 +CPU= 18.183
  02:37:27 STEP 2825 H=2825:00 +CPU= 17.568
  02:37:57 STEP 2826 H=2826:00 +CPU= 29.434
  02:38:15 STEP 2827 H=2827:00 +CPU= 17.897
  02:38:34 STEP 2828 H=2828:00 +CPU= 18.912
  02:39:03 STEP 2829 H=2829:00 +CPU= 29.175
  02:39:21 STEP 2830 H=2830:00 +CPU= 18.065
  02:39:40 STEP 2831 H=2831:00 +CPU= 18.842
  02:40:16 STEP 2832 H=2832:00 +CPU= 30.890
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002832
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002832
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002808
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002820
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002820
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_117.zip
Uploading the intermediate file: upload_file_117.zip
  02:40:33 STEP 2833 H=2833:00 +CPU= 17.250
  02:40:51 STEP 2834 H=2834:00 +CPU= 17.889
  02:41:19 STEP 2835 H=2835:00 +CPU= 28.353
  02:41:36 STEP 2836 H=2836:00 +CPU= 17.240
  02:41:54 STEP 2837 H=2837:00 +CPU= 17.505
  02:42:21 STEP 2838 H=2838:00 +CPU= 26.744
  02:42:38 STEP 2839 H=2839:00 +CPU= 17.602
  02:42:55 STEP 2840 H=2840:00 +CPU= 16.773
  02:43:21 STEP 2841 H=2841:00 +CPU= 26.126
  02:43:38 STEP 2842 H=2842:00 +CPU= 17.237
  02:43:56 STEP 2843 H=2843:00 +CPU= 17.590
  02:44:26 STEP 2844 H=2844:00 +CPU= 30.140
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002844
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002844
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002844
  02:44:44 STEP 2845 H=2845:00 +CPU= 18.325
  02:45:02 STEP 2846 H=2846:00 +CPU= 17.526
  02:45:31 STEP 2847 H=2847:00 +CPU= 29.067
  02:45:49 STEP 2848 H=2848:00 +CPU= 17.927
  02:46:08 STEP 2849 H=2849:00 +CPU= 18.696
  02:46:36 STEP 2850 H=2850:00 +CPU= 28.134
  02:46:54 STEP 2851 H=2851:00 +CPU= 17.746
  02:47:12 STEP 2852 H=2852:00 +CPU= 18.761
  02:47:40 STEP 2853 H=2853:00 +CPU= 27.681
  02:48:00 STEP 2854 H=2854:00 +CPU= 18.331
  02:48:20 STEP 2855 H=2855:00 +CPU= 20.024
  02:48:56 STEP 2856 H=2856:00 +CPU= 31.667
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002856
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002856
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002832
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002844
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002844
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_118.zip
Uploading the intermediate file: upload_file_118.zip
  02:49:13 STEP 2857 H=2857:00 +CPU= 17.260
  02:49:31 STEP 2858 H=2858:00 +CPU= 17.766
  02:49:58 STEP 2859 H=2859:00 +CPU= 27.748
  02:50:16 STEP 2860 H=2860:00 +CPU= 18.018
  02:50:33 STEP 2861 H=2861:00 +CPU= 16.738
  02:51:03 STEP 2862 H=2862:00 +CPU= 29.746
  02:51:20 STEP 2863 H=2863:00 +CPU= 16.615
  02:51:38 STEP 2864 H=2864:00 +CPU= 17.185
  02:52:07 STEP 2865 H=2865:00 +CPU= 29.605
  02:52:26 STEP 2866 H=2866:00 +CPU= 18.348
  02:52:43 STEP 2867 H=2867:00 +CPU= 17.160
  02:53:15 STEP 2868 H=2868:00 +CPU= 32.380
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002868
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002868
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002868
  02:53:32 STEP 2869 H=2869:00 +CPU= 16.481
  02:53:49 STEP 2870 H=2870:00 +CPU= 17.181
  02:54:16 STEP 2871 H=2871:00 +CPU= 26.620
  02:54:34 STEP 2872 H=2872:00 +CPU= 18.149
  02:54:51 STEP 2873 H=2873:00 +CPU= 17.371
  02:55:19 STEP 2874 H=2874:00 +CPU= 28.371
  02:55:38 STEP 2875 H=2875:00 +CPU= 18.942
  02:55:58 STEP 2876 H=2876:00 +CPU= 19.553
  02:56:26 STEP 2877 H=2877:00 +CPU= 28.046
  02:56:44 STEP 2878 H=2878:00 +CPU= 18.410
  02:57:03 STEP 2879 H=2879:00 +CPU= 18.642
  02:57:37 STEP 2880 H=2880:00 +CPU= 33.302
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002880
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002880
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002856
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002868
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002868
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_119.zip
Uploading the intermediate file: upload_file_119.zip
  02:57:54 STEP 2881 H=2881:00 +CPU= 16.976
  02:58:12 STEP 2882 H=2882:00 +CPU= 17.760
  02:58:40 STEP 2883 H=2883:00 +CPU= 27.840
  02:58:58 STEP 2884 H=2884:00 +CPU= 18.245
  02:59:15 STEP 2885 H=2885:00 +CPU= 16.881
  02:59:44 STEP 2886 H=2886:00 +CPU= 27.539
  03:00:01 STEP 2887 H=2887:00 +CPU= 17.448
  03:00:19 STEP 2888 H=2888:00 +CPU= 17.877
  03:00:47 STEP 2889 H=2889:00 +CPU= 27.833
  03:01:05 STEP 2890 H=2890:00 +CPU= 18.500
  03:01:23 STEP 2891 H=2891:00 +CPU= 17.523
  03:01:54 STEP 2892 H=2892:00 +CPU= 31.049
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002892
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002892
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002892
  03:02:11 STEP 2893 H=2893:00 +CPU= 17.506
  03:02:29 STEP 2894 H=2894:00 +CPU= 17.945
  03:02:57 STEP 2895 H=2895:00 +CPU= 28.107
  03:03:15 STEP 2896 H=2896:00 +CPU= 18.053
  03:03:33 STEP 2897 H=2897:00 +CPU= 17.157
  03:04:00 STEP 2898 H=2898:00 +CPU= 27.543
  03:04:16 STEP 2899 H=2899:00 +CPU= 16.274
  03:04:34 STEP 2900 H=2900:00 +CPU= 17.186
  03:05:01 STEP 2901 H=2901:00 +CPU= 26.824
  03:05:17 STEP 2902 H=2902:00 +CPU= 16.666
  03:05:35 STEP 2903 H=2903:00 +CPU= 17.333
  03:06:16 STEP 2904 H=2904:00 +CPU= 33.266
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002904
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002904
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002880
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002892
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002892
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_120.zip
Uploading the intermediate file: upload_file_120.zip
  03:06:35 STEP 2905 H=2905:00 +CPU= 18.080
  03:06:52 STEP 2906 H=2906:00 +CPU= 17.211
  03:07:19 STEP 2907 H=2907:00 +CPU= 27.250
  03:07:37 STEP 2908 H=2908:00 +CPU= 18.488
  03:07:54 STEP 2909 H=2909:00 +CPU= 16.222
  03:08:21 STEP 2910 H=2910:00 +CPU= 26.874
  03:08:38 STEP 2911 H=2911:00 +CPU= 17.763
  03:08:56 STEP 2912 H=2912:00 +CPU= 17.202
  03:09:22 STEP 2913 H=2913:00 +CPU= 25.335
  03:09:39 STEP 2914 H=2914:00 +CPU= 17.566
  03:09:56 STEP 2915 H=2915:00 +CPU= 16.753
  03:10:26 STEP 2916 H=2916:00 +CPU= 29.879
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002916
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002916
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002916
  03:10:48 STEP 2917 H=2917:00 +CPU= 16.346
  03:11:05 STEP 2918 H=2918:00 +CPU= 17.343
  03:11:32 STEP 2919 H=2919:00 +CPU= 26.469
  03:11:49 STEP 2920 H=2920:00 +CPU= 17.723
  03:12:07 STEP 2921 H=2921:00 +CPU= 17.156
  03:12:33 STEP 2922 H=2922:00 +CPU= 26.693
  03:12:50 STEP 2923 H=2923:00 +CPU= 16.626
  03:13:08 STEP 2924 H=2924:00 +CPU= 17.609
  03:13:34 STEP 2925 H=2925:00 +CPU= 26.875
  03:13:52 STEP 2926 H=2926:00 +CPU= 17.510
  03:14:09 STEP 2927 H=2927:00 +CPU= 16.846
  03:14:46 STEP 2928 H=2928:00 +CPU= 31.454
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002928
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002928
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002904
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002916
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002916
Zipping up the intermediate file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_121.zip
Uploading the intermediate file: upload_file_121.zip
  03:15:03 STEP 2929 H=2929:00 +CPU= 17.239
  03:15:36 STEP 2930 H=2930:00 +CPU= 17.717
  03:16:02 STEP 2931 H=2931:00 +CPU= 25.918
  03:16:20 STEP 2932 H=2932:00 +CPU= 17.659
  03:16:36 STEP 2933 H=2933:00 +CPU= 16.607
  03:17:03 STEP 2934 H=2934:00 +CPU= 26.667
  03:17:19 STEP 2935 H=2935:00 +CPU= 16.539
  03:17:36 STEP 2936 H=2936:00 +CPU= 16.990
  03:18:02 STEP 2937 H=2937:00 +CPU= 26.133
  03:18:20 STEP 2938 H=2938:00 +CPU= 17.326
  03:18:36 STEP 2939 H=2939:00 +CPU= 16.536
  03:19:06 STEP 2940 H=2940:00 +CPU= 29.881
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002940
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002940
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002940
  03:19:24 STEP 2941 H=2941:00 +CPU= 17.351
  03:19:41 STEP 2942 H=2942:00 +CPU= 17.146
  03:20:08 STEP 2943 H=2943:00 +CPU= 26.709
  03:20:25 STEP 2944 H=2944:00 +CPU= 17.359
  03:20:42 STEP 2945 H=2945:00 +CPU= 17.101
  03:21:09 STEP 2946 H=2946:00 +CPU= 27.308
  03:21:26 STEP 2947 H=2947:00 +CPU= 16.393
  03:21:43 STEP 2948 H=2948:00 +CPU= 16.807
  03:22:09 STEP 2949 H=2949:00 +CPU= 26.473
  03:22:26 STEP 2950 H=2950:00 +CPU= 16.620
  03:22:42 STEP 2951 H=2951:00 +CPU= 16.529
  03:23:15 STEP 2952 H=2952:00 +CPU= 29.910
The child process terminated with status: 0
Moving to projects directory: /home/gpuser/slots/3/ICMGGhq0f+002952
Moving to projects directory: /home/gpuser/slots/3/ICMSHhq0f+002952
Moving to projects directory: /home/gpuser/slots/3/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002928
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMGGhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMSHhq0f+002940
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002952
Adding to the zip: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_12187621/ICMUAhq0f+002928
Zipping up the final file: /home/gpuser/projects/climateprediction.net/oifs_43r3_ps_0977_2001050100_123_970_12187621_0_r462979640_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
03:26:39 (54): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:09:13 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 10,623,600 48,952 0.0046
09 May 2023 16:09:09 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 10,022,400 45,436 0.0045
09 May 2023 16:09:05 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 9,158,400 40,567 0.0044
09 May 2023 16:09:02 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 8,294,400 35,645 0.0043
09 May 2023 16:08:54 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 7,430,400 30,572 0.0041
09 May 2023 16:08:50 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 6,566,400 25,460 0.0039
09 May 2023 16:08:46 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 5,702,400 20,402 0.0036
09 May 2023 16:08:43 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 4,838,400 17,311 0.0036
09 May 2023 16:08:39 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 3,974,400 14,392 0.0036
09 May 2023 16:08:36 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 3,110,400 11,562 0.0037
09 May 2023 16:08:27 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 1,728,000 6,422 0.0037
09 May 2023 16:08:27 1537625 22273057 oifs_43r3_ps_0977_2001050100_123_970_12187621_0 864,000 3,466 0.0040


©2024 climateprediction.net