climateprediction.net home page
Task 22252323

Task 22252323

Name oifs_43r3_ps_0438_1982050100_123_951_12168082_0
Workunit 12168082
Created 20 Dec 2022, 22:08:37 UTC
Sent 12 Jan 2023, 0:18:29 UTC
Report deadline 11 Feb 2023, 0:18:29 UTC
Received 14 Jan 2023, 9:13:17 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1545600
Run time 1 days 8 hours 47 min 43 sec
CPU time 23 hours 11 min 50 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 6.17 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,410.91 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,329.70 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
diate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_98.zip
Uploading the intermediate file: upload_file_98.zip
  22:15:24 STEP 2377 H=2377:00 +CPU= 23.743
  22:15:49 STEP 2378 H=2378:00 +CPU= 22.661
  22:16:30 STEP 2379 H=2379:00 +CPU= 34.316
  22:16:58 STEP 2380 H=2380:00 +CPU= 23.162
  22:17:23 STEP 2381 H=2381:00 +CPU= 22.376
  22:18:02 STEP 2382 H=2382:00 +CPU= 35.206
  22:18:25 STEP 2383 H=2383:00 +CPU= 21.382
  22:18:52 STEP 2384 H=2384:00 +CPU= 23.440
  22:19:31 STEP 2385 H=2385:00 +CPU= 34.645
  22:19:58 STEP 2386 H=2386:00 +CPU= 23.725
  22:20:28 STEP 2387 H=2387:00 +CPU= 23.141
  22:21:13 STEP 2388 H=2388:00 +CPU= 38.101
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002388
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002388
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002388
  22:21:39 STEP 2389 H=2389:00 +CPU= 22.492
  22:22:06 STEP 2390 H=2390:00 +CPU= 21.829
  22:22:42 STEP 2391 H=2391:00 +CPU= 33.051
  22:23:08 STEP 2392 H=2392:00 +CPU= 21.661
  22:23:34 STEP 2393 H=2393:00 +CPU= 22.546
  22:24:17 STEP 2394 H=2394:00 +CPU= 37.058
  22:24:44 STEP 2395 H=2395:00 +CPU= 23.564
  22:25:12 STEP 2396 H=2396:00 +CPU= 24.566
  22:25:55 STEP 2397 H=2397:00 +CPU= 35.580
  22:26:21 STEP 2398 H=2398:00 +CPU= 22.682
  22:26:47 STEP 2399 H=2399:00 +CPU= 23.305
  22:27:37 STEP 2400 H=2400:00 +CPU= 41.429
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002400
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002400
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002376
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002376
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002376
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002388
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002388
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002388
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_99.zip
Uploading the intermediate file: upload_file_99.zip
  22:28:04 STEP 2401 H=2401:00 +CPU= 23.220
Uploading trickle at timestep: 8640000
  22:28:28 STEP 2402 H=2402:00 +CPU= 22.385
  22:29:06 STEP 2403 H=2403:00 +CPU= 32.368
  22:29:33 STEP 2404 H=2404:00 +CPU= 21.994
  22:29:57 STEP 2405 H=2405:00 +CPU= 20.827
  22:30:41 STEP 2406 H=2406:00 +CPU= 36.548
  22:31:07 STEP 2407 H=2407:00 +CPU= 22.765
  22:31:33 STEP 2408 H=2408:00 +CPU= 23.007
  22:32:12 STEP 2409 H=2409:00 +CPU= 34.570
  22:32:36 STEP 2410 H=2410:00 +CPU= 22.006
  22:33:01 STEP 2411 H=2411:00 +CPU= 22.926
  22:33:50 STEP 2412 H=2412:00 +CPU= 39.600
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002412
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002412
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002412
  22:34:15 STEP 2413 H=2413:00 +CPU= 22.113
  22:34:40 STEP 2414 H=2414:00 +CPU= 22.420
  22:35:19 STEP 2415 H=2415:00 +CPU= 34.681
  22:35:43 STEP 2416 H=2416:00 +CPU= 21.538
  22:36:05 STEP 2417 H=2417:00 +CPU= 20.187
  22:36:38 STEP 2418 H=2418:00 +CPU= 32.041
  22:37:04 STEP 2419 H=2419:00 +CPU= 21.620
  22:37:35 STEP 2420 H=2420:00 +CPU= 23.025
  22:38:11 STEP 2421 H=2421:00 +CPU= 33.288
  22:38:36 STEP 2422 H=2422:00 +CPU= 22.747
  22:39:05 STEP 2423 H=2423:00 +CPU= 23.775
  22:39:53 STEP 2424 H=2424:00 +CPU= 39.694
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002424
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002424
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002412
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002412
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002412
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_100.zip
Uploading the intermediate file: upload_file_100.zip
  22:40:19 STEP 2425 H=2425:00 +CPU= 22.657
  22:40:45 STEP 2426 H=2426:00 +CPU= 23.229
  22:41:25 STEP 2427 H=2427:00 +CPU= 34.546
  22:41:51 STEP 2428 H=2428:00 +CPU= 23.731
  22:42:16 STEP 2429 H=2429:00 +CPU= 22.604
  22:42:55 STEP 2430 H=2430:00 +CPU= 33.945
  22:43:19 STEP 2431 H=2431:00 +CPU= 21.371
  22:43:50 STEP 2432 H=2432:00 +CPU= 24.382
  22:44:28 STEP 2433 H=2433:00 +CPU= 35.245
  22:44:57 STEP 2434 H=2434:00 +CPU= 24.257
  22:45:24 STEP 2435 H=2435:00 +CPU= 21.447
  22:46:04 STEP 2436 H=2436:00 +CPU= 36.406
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002436
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002436
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002436
  22:46:29 STEP 2437 H=2437:00 +CPU= 22.705
  22:46:54 STEP 2438 H=2438:00 +CPU= 22.657
  22:47:36 STEP 2439 H=2439:00 +CPU= 35.078
  22:48:00 STEP 2440 H=2440:00 +CPU= 21.397
  22:48:23 STEP 2441 H=2441:00 +CPU= 21.331
  22:49:01 STEP 2442 H=2442:00 +CPU= 34.547
  22:49:25 STEP 2443 H=2443:00 +CPU= 21.686
  22:49:51 STEP 2444 H=2444:00 +CPU= 23.565
  22:50:32 STEP 2445 H=2445:00 +CPU= 35.139
  22:50:58 STEP 2446 H=2446:00 +CPU= 23.229
  22:51:25 STEP 2447 H=2447:00 +CPU= 22.780
  22:52:12 STEP 2448 H=2448:00 +CPU= 39.977
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002448
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002448
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002436
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002436
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002436
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_101.zip
Uploading the intermediate file: upload_file_101.zip
  22:52:39 STEP 2449 H=2449:00 +CPU= 23.403
  22:53:06 STEP 2450 H=2450:00 +CPU= 23.070
  22:53:44 STEP 2451 H=2451:00 +CPU= 34.044
  22:54:11 STEP 2452 H=2452:00 +CPU= 24.210
  22:54:37 STEP 2453 H=2453:00 +CPU= 22.557
  22:55:18 STEP 2454 H=2454:00 +CPU= 36.879
  22:55:42 STEP 2455 H=2455:00 +CPU= 21.439
  22:56:10 STEP 2456 H=2456:00 +CPU= 22.830
  22:56:53 STEP 2457 H=2457:00 +CPU= 37.450
  22:57:18 STEP 2458 H=2458:00 +CPU= 21.943
  22:57:43 STEP 2459 H=2459:00 +CPU= 23.004
  22:58:26 STEP 2460 H=2460:00 +CPU= 37.512
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002460
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002460
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002460
  22:58:50 STEP 2461 H=2461:00 +CPU= 22.360
  22:59:19 STEP 2462 H=2462:00 +CPU= 23.555
  22:59:59 STEP 2463 H=2463:00 +CPU= 35.365
  23:00:25 STEP 2464 H=2464:00 +CPU= 22.401
  23:00:52 STEP 2465 H=2465:00 +CPU= 21.635
  23:01:33 STEP 2466 H=2466:00 +CPU= 36.844
  23:01:58 STEP 2467 H=2467:00 +CPU= 21.594
  23:02:26 STEP 2468 H=2468:00 +CPU= 25.030
  23:03:08 STEP 2469 H=2469:00 +CPU= 34.979
  23:03:35 STEP 2470 H=2470:00 +CPU= 23.938
  23:03:59 STEP 2471 H=2471:00 +CPU= 22.651
  23:04:44 STEP 2472 H=2472:00 +CPU= 40.785
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002472
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002472
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002460
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002460
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002460
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_102.zip
Uploading the intermediate file: upload_file_102.zip
  23:05:09 STEP 2473 H=2473:00 +CPU= 21.529
  23:05:35 STEP 2474 H=2474:00 +CPU= 22.485
  23:06:13 STEP 2475 H=2475:00 +CPU= 33.493
  23:06:39 STEP 2476 H=2476:00 +CPU= 22.403
  23:07:05 STEP 2477 H=2477:00 +CPU= 21.019
  23:07:48 STEP 2478 H=2478:00 +CPU= 36.631
  23:08:15 STEP 2479 H=2479:00 +CPU= 22.238
  23:08:39 STEP 2480 H=2480:00 +CPU= 21.262
  23:09:15 STEP 2481 H=2481:00 +CPU= 33.083
  23:09:41 STEP 2482 H=2482:00 +CPU= 23.074
  23:10:06 STEP 2483 H=2483:00 +CPU= 22.273
  23:10:52 STEP 2484 H=2484:00 +CPU= 37.961
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002484
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002484
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002484
  23:11:16 STEP 2485 H=2485:00 +CPU= 21.996
  23:11:40 STEP 2486 H=2486:00 +CPU= 20.921
  23:12:19 STEP 2487 H=2487:00 +CPU= 33.729
  23:12:48 STEP 2488 H=2488:00 +CPU= 23.089
  23:13:12 STEP 2489 H=2489:00 +CPU= 21.175
  23:13:50 STEP 2490 H=2490:00 +CPU= 34.384
  23:14:17 STEP 2491 H=2491:00 +CPU= 22.509
  23:14:43 STEP 2492 H=2492:00 +CPU= 22.960
  23:15:18 STEP 2493 H=2493:00 +CPU= 32.244
  23:15:42 STEP 2494 H=2494:00 +CPU= 22.417
  23:16:08 STEP 2495 H=2495:00 +CPU= 22.633
  23:16:54 STEP 2496 H=2496:00 +CPU= 40.519
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002496
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002496
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002484
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002484
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002484
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_103.zip
Uploading the intermediate file: upload_file_103.zip
  23:17:15 STEP 2497 H=2497:00 +CPU= 20.579
  23:17:41 STEP 2498 H=2498:00 +CPU= 23.545
  23:18:19 STEP 2499 H=2499:00 +CPU= 32.715
  23:18:46 STEP 2500 H=2500:00 +CPU= 23.213
  23:19:09 STEP 2501 H=2501:00 +CPU= 21.502
  23:19:49 STEP 2502 H=2502:00 +CPU= 35.028
  23:20:13 STEP 2503 H=2503:00 +CPU= 21.437
  23:20:40 STEP 2504 H=2504:00 +CPU= 22.767
  23:21:24 STEP 2505 H=2505:00 +CPU= 35.897
  23:21:45 STEP 2506 H=2506:00 +CPU= 19.984
  23:22:10 STEP 2507 H=2507:00 +CPU= 21.489
  23:22:54 STEP 2508 H=2508:00 +CPU= 38.561
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002508
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002508
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002508
  23:23:20 STEP 2509 H=2509:00 +CPU= 21.105
  23:23:48 STEP 2510 H=2510:00 +CPU= 23.643
  23:24:24 STEP 2511 H=2511:00 +CPU= 32.892
  23:24:49 STEP 2512 H=2512:00 +CPU= 21.867
  23:25:17 STEP 2513 H=2513:00 +CPU= 22.451
  23:25:57 STEP 2514 H=2514:00 +CPU= 34.698
  23:26:25 STEP 2515 H=2515:00 +CPU= 22.068
  23:26:54 STEP 2516 H=2516:00 +CPU= 22.059
  23:27:33 STEP 2517 H=2517:00 +CPU= 33.593
  23:27:59 STEP 2518 H=2518:00 +CPU= 23.277
  23:28:28 STEP 2519 H=2519:00 +CPU= 22.376
  23:29:14 STEP 2520 H=2520:00 +CPU= 40.908
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002520
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002520
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002508
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002508
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002508
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_104.zip
Uploading the intermediate file: upload_file_104.zip
  23:29:37 STEP 2521 H=2521:00 +CPU= 20.349
  23:30:00 STEP 2522 H=2522:00 +CPU= 20.800
  23:30:37 STEP 2523 H=2523:00 +CPU= 33.998
  23:31:03 STEP 2524 H=2524:00 +CPU= 23.591
  23:31:30 STEP 2525 H=2525:00 +CPU= 22.674
  23:32:10 STEP 2526 H=2526:00 +CPU= 34.871
  23:32:34 STEP 2527 H=2527:00 +CPU= 21.592
  23:32:58 STEP 2528 H=2528:00 +CPU= 22.651
  23:33:38 STEP 2529 H=2529:00 +CPU= 34.210
  23:34:04 STEP 2530 H=2530:00 +CPU= 22.533
  23:34:29 STEP 2531 H=2531:00 +CPU= 21.088
  23:35:10 STEP 2532 H=2532:00 +CPU= 37.186
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002532
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002532
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002532
  23:35:34 STEP 2533 H=2533:00 +CPU= 22.040
  23:36:00 STEP 2534 H=2534:00 +CPU= 23.428
  23:36:39 STEP 2535 H=2535:00 +CPU= 34.491
  23:37:04 STEP 2536 H=2536:00 +CPU= 23.345
  23:37:27 STEP 2537 H=2537:00 +CPU= 21.331
  23:38:05 STEP 2538 H=2538:00 +CPU= 35.326
  23:38:31 STEP 2539 H=2539:00 +CPU= 21.581
  23:38:58 STEP 2540 H=2540:00 +CPU= 22.989
  23:39:35 STEP 2541 H=2541:00 +CPU= 33.739
  23:39:59 STEP 2542 H=2542:00 +CPU= 21.871
  23:40:22 STEP 2543 H=2543:00 +CPU= 21.369
  23:41:07 STEP 2544 H=2544:00 +CPU= 39.731
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002544
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002544
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002532
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002532
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002532
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_105.zip
Uploading the intermediate file: upload_file_105.zip
  23:41:30 STEP 2545 H=2545:00 +CPU= 21.161
  23:41:53 STEP 2546 H=2546:00 +CPU= 21.390
  23:42:34 STEP 2547 H=2547:00 +CPU= 35.510
  23:43:01 STEP 2548 H=2548:00 +CPU= 22.734
  23:43:29 STEP 2549 H=2549:00 +CPU= 24.230
  23:44:10 STEP 2550 H=2550:00 +CPU= 36.157
  23:44:37 STEP 2551 H=2551:00 +CPU= 23.197
  23:45:08 STEP 2552 H=2552:00 +CPU= 24.647
  23:45:46 STEP 2553 H=2553:00 +CPU= 34.670
  23:46:14 STEP 2554 H=2554:00 +CPU= 23.394
  23:46:39 STEP 2555 H=2555:00 +CPU= 23.741
  23:47:20 STEP 2556 H=2556:00 +CPU= 38.558
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002556
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002556
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002556
  23:47:47 STEP 2557 H=2557:00 +CPU= 23.137
  23:48:15 STEP 2558 H=2558:00 +CPU= 22.005
  23:48:51 STEP 2559 H=2559:00 +CPU= 33.110
  23:49:16 STEP 2560 H=2560:00 +CPU= 22.560
  23:49:40 STEP 2561 H=2561:00 +CPU= 21.999
  23:50:19 STEP 2562 H=2562:00 +CPU= 35.438
  23:50:45 STEP 2563 H=2563:00 +CPU= 22.992
  23:51:13 STEP 2564 H=2564:00 +CPU= 24.349
  23:51:55 STEP 2565 H=2565:00 +CPU= 36.292
  23:52:20 STEP 2566 H=2566:00 +CPU= 22.774
  23:52:48 STEP 2567 H=2567:00 +CPU= 24.056
  23:53:36 STEP 2568 H=2568:00 +CPU= 43.493
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002568
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002568
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002556
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002556
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002556
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_106.zip
Uploading the intermediate file: upload_file_106.zip
  23:54:01 STEP 2569 H=2569:00 +CPU= 22.913
  23:54:30 STEP 2570 H=2570:00 +CPU= 23.270
  23:55:10 STEP 2571 H=2571:00 +CPU= 34.815
  23:55:33 STEP 2572 H=2572:00 +CPU= 21.779
  23:55:57 STEP 2573 H=2573:00 +CPU= 21.279
  23:56:37 STEP 2574 H=2574:00 +CPU= 35.016
  23:57:01 STEP 2575 H=2575:00 +CPU= 20.448
  23:57:26 STEP 2576 H=2576:00 +CPU= 22.122
  23:58:08 STEP 2577 H=2577:00 +CPU= 34.858
  23:58:31 STEP 2578 H=2578:00 +CPU= 20.982
  23:58:54 STEP 2579 H=2579:00 +CPU= 21.294
  23:59:36 STEP 2580 H=2580:00 +CPU= 37.764
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002580
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002580
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002580
  00:00:03 STEP 2581 H=2581:00 +CPU= 23.750
  00:00:40 STEP 2582 H=2582:00 +CPU= 25.839
  00:01:19 STEP 2583 H=2583:00 +CPU= 34.971
  00:01:45 STEP 2584 H=2584:00 +CPU= 21.785
  00:02:15 STEP 2585 H=2585:00 +CPU= 24.035
  00:02:57 STEP 2586 H=2586:00 +CPU= 35.731
  00:03:25 STEP 2587 H=2587:00 +CPU= 23.188
  00:03:54 STEP 2588 H=2588:00 +CPU= 24.518
  00:04:35 STEP 2589 H=2589:00 +CPU= 36.573
  00:05:02 STEP 2590 H=2590:00 +CPU= 22.906
  00:05:28 STEP 2591 H=2591:00 +CPU= 23.864
  00:06:12 STEP 2592 H=2592:00 +CPU= 39.305
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002592
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002592
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002580
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002580
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002580
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_107.zip
Uploading the intermediate file: upload_file_107.zip
  00:06:38 STEP 2593 H=2593:00 +CPU= 22.557
  00:07:06 STEP 2594 H=2594:00 +CPU= 23.181
  00:07:47 STEP 2595 H=2595:00 +CPU= 34.669
  00:08:16 STEP 2596 H=2596:00 +CPU= 22.996
  00:08:41 STEP 2597 H=2597:00 +CPU= 22.049
  00:09:20 STEP 2598 H=2598:00 +CPU= 34.035
  00:09:44 STEP 2599 H=2599:00 +CPU= 22.646
  00:10:12 STEP 2600 H=2600:00 +CPU= 23.504
  00:10:50 STEP 2601 H=2601:00 +CPU= 33.835
  00:11:16 STEP 2602 H=2602:00 +CPU= 22.452
  00:11:42 STEP 2603 H=2603:00 +CPU= 24.089
  00:12:29 STEP 2604 H=2604:00 +CPU= 39.119
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002604
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002604
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002604
  00:12:53 STEP 2605 H=2605:00 +CPU= 21.700
  00:13:21 STEP 2606 H=2606:00 +CPU= 23.482
  00:14:01 STEP 2607 H=2607:00 +CPU= 35.140
  00:14:27 STEP 2608 H=2608:00 +CPU= 22.397
  00:14:52 STEP 2609 H=2609:00 +CPU= 22.430
  00:15:31 STEP 2610 H=2610:00 +CPU= 34.119
  00:15:59 STEP 2611 H=2611:00 +CPU= 22.564
  00:16:24 STEP 2612 H=2612:00 +CPU= 22.727
  00:17:03 STEP 2613 H=2613:00 +CPU= 33.136
  00:17:33 STEP 2614 H=2614:00 +CPU= 23.125
  00:17:57 STEP 2615 H=2615:00 +CPU= 21.846
  00:18:44 STEP 2616 H=2616:00 +CPU= 40.059
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002616
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002616
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002604
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002604
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002604
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_108.zip
Uploading the intermediate file: upload_file_108.zip
  00:19:08 STEP 2617 H=2617:00 +CPU= 21.757
  00:19:34 STEP 2618 H=2618:00 +CPU= 21.986
  00:20:05 STEP 2619 H=2619:00 +CPU= 30.964
  00:20:33 STEP 2620 H=2620:00 +CPU= 23.182
  00:21:00 STEP 2621 H=2621:00 +CPU= 22.512
  00:21:40 STEP 2622 H=2622:00 +CPU= 34.638
  00:22:06 STEP 2623 H=2623:00 +CPU= 22.667
  00:22:28 STEP 2624 H=2624:00 +CPU= 20.601
  00:23:01 STEP 2625 H=2625:00 +CPU= 30.642
  00:23:30 STEP 2626 H=2626:00 +CPU= 22.739
  00:23:58 STEP 2627 H=2627:00 +CPU= 22.409
  00:24:43 STEP 2628 H=2628:00 +CPU= 36.998
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002628
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002628
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002628
  00:25:10 STEP 2629 H=2629:00 +CPU= 22.493
  00:25:41 STEP 2630 H=2630:00 +CPU= 23.127
  00:26:20 STEP 2631 H=2631:00 +CPU= 34.721
  00:26:49 STEP 2632 H=2632:00 +CPU= 23.985
  00:27:15 STEP 2633 H=2633:00 +CPU= 22.388
  00:27:52 STEP 2634 H=2634:00 +CPU= 32.652
  00:28:15 STEP 2635 H=2635:00 +CPU= 20.452
  00:28:37 STEP 2636 H=2636:00 +CPU= 21.214
  00:29:17 STEP 2637 H=2637:00 +CPU= 33.642
  00:29:42 STEP 2638 H=2638:00 +CPU= 22.174
  00:30:07 STEP 2639 H=2639:00 +CPU= 20.642
  00:30:55 STEP 2640 H=2640:00 +CPU= 40.510
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002640
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002640
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002628
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002628
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002628
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_109.zip
Uploading the intermediate file: upload_file_109.zip
  00:31:18 STEP 2641 H=2641:00 +CPU= 21.578
Uploading trickle at timestep: 9504000
  00:31:48 STEP 2642 H=2642:00 +CPU= 23.526
  00:32:28 STEP 2643 H=2643:00 +CPU= 33.270
  00:32:57 STEP 2644 H=2644:00 +CPU= 24.697
  00:33:21 STEP 2645 H=2645:00 +CPU= 21.230
  00:34:04 STEP 2646 H=2646:00 +CPU= 34.619
  00:34:29 STEP 2647 H=2647:00 +CPU= 22.843
  00:34:57 STEP 2648 H=2648:00 +CPU= 22.946
  00:35:36 STEP 2649 H=2649:00 +CPU= 34.813
  00:36:04 STEP 2650 H=2650:00 +CPU= 23.979
  00:36:29 STEP 2651 H=2651:00 +CPU= 19.898
  00:37:07 STEP 2652 H=2652:00 +CPU= 35.416
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002652
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002652
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002652
  00:37:28 STEP 2653 H=2653:00 +CPU= 20.659
  00:37:53 STEP 2654 H=2654:00 +CPU= 21.965
  00:38:33 STEP 2655 H=2655:00 +CPU= 33.285
  00:39:01 STEP 2656 H=2656:00 +CPU= 22.675
  00:39:29 STEP 2657 H=2657:00 +CPU= 22.777
  00:40:11 STEP 2658 H=2658:00 +CPU= 36.438
  00:40:37 STEP 2659 H=2659:00 +CPU= 22.451
  00:41:03 STEP 2660 H=2660:00 +CPU= 22.494
  00:41:45 STEP 2661 H=2661:00 +CPU= 34.324
  00:42:08 STEP 2662 H=2662:00 +CPU= 20.936
  00:42:36 STEP 2663 H=2663:00 +CPU= 23.069
  00:43:26 STEP 2664 H=2664:00 +CPU= 41.408
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002664
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002664
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002652
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002652
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002652
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_110.zip
Uploading the intermediate file: upload_file_110.zip
  00:43:55 STEP 2665 H=2665:00 +CPU= 22.544
  00:44:21 STEP 2666 H=2666:00 +CPU= 22.159
  00:44:56 STEP 2667 H=2667:00 +CPU= 31.147
  00:45:23 STEP 2668 H=2668:00 +CPU= 22.438
  00:45:48 STEP 2669 H=2669:00 +CPU= 21.935
  00:46:27 STEP 2670 H=2670:00 +CPU= 35.015
  00:46:49 STEP 2671 H=2671:00 +CPU= 20.245
  00:47:16 STEP 2672 H=2672:00 +CPU= 23.120
  00:47:57 STEP 2673 H=2673:00 +CPU= 34.280
  00:48:21 STEP 2674 H=2674:00 +CPU= 22.156
  00:48:43 STEP 2675 H=2675:00 +CPU= 20.549
  00:49:25 STEP 2676 H=2676:00 +CPU= 36.487
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002676
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002676
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002676
  00:49:49 STEP 2677 H=2677:00 +CPU= 21.807
  00:50:14 STEP 2678 H=2678:00 +CPU= 22.852
  00:50:51 STEP 2679 H=2679:00 +CPU= 32.476
  00:51:14 STEP 2680 H=2680:00 +CPU= 22.912
  00:51:41 STEP 2681 H=2681:00 +CPU= 22.153
  00:52:21 STEP 2682 H=2682:00 +CPU= 34.485
  00:52:44 STEP 2683 H=2683:00 +CPU= 22.491
  00:53:09 STEP 2684 H=2684:00 +CPU= 21.627
  00:53:50 STEP 2685 H=2685:00 +CPU= 34.842
  00:54:19 STEP 2686 H=2686:00 +CPU= 24.134
  00:54:44 STEP 2687 H=2687:00 +CPU= 21.531
  00:55:32 STEP 2688 H=2688:00 +CPU= 41.726
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002688
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002688
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002676
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002676
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002676
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_111.zip
Uploading the intermediate file: upload_file_111.zip
  00:55:57 STEP 2689 H=2689:00 +CPU= 22.108
  00:56:26 STEP 2690 H=2690:00 +CPU= 22.954
  00:57:05 STEP 2691 H=2691:00 +CPU= 34.031
  00:57:30 STEP 2692 H=2692:00 +CPU= 22.942
  00:57:54 STEP 2693 H=2693:00 +CPU= 22.601
  00:58:35 STEP 2694 H=2694:00 +CPU= 35.611
  00:59:06 STEP 2695 H=2695:00 +CPU= 23.209
  00:59:32 STEP 2696 H=2696:00 +CPU= 22.359
  01:00:07 STEP 2697 H=2697:00 +CPU= 31.581
  01:00:32 STEP 2698 H=2698:00 +CPU= 21.816
  01:00:59 STEP 2699 H=2699:00 +CPU= 22.845
  01:01:50 STEP 2700 H=2700:00 +CPU= 39.979
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002700
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002700
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002700
  01:02:18 STEP 2701 H=2701:00 +CPU= 22.437
  01:02:45 STEP 2702 H=2702:00 +CPU= 22.653
  01:03:23 STEP 2703 H=2703:00 +CPU= 34.160
  01:03:48 STEP 2704 H=2704:00 +CPU= 22.399
  01:04:15 STEP 2705 H=2705:00 +CPU= 22.735
  01:04:52 STEP 2706 H=2706:00 +CPU= 33.589
  01:05:14 STEP 2707 H=2707:00 +CPU= 19.931
  01:05:38 STEP 2708 H=2708:00 +CPU= 21.381
  01:06:18 STEP 2709 H=2709:00 +CPU= 35.516
  01:06:44 STEP 2710 H=2710:00 +CPU= 22.617
  01:07:12 STEP 2711 H=2711:00 +CPU= 22.815
  01:08:03 STEP 2712 H=2712:00 +CPU= 43.078
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002712
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002712
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002700
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002700
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002700
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_112.zip
Uploading the intermediate file: upload_file_112.zip
  01:08:30 STEP 2713 H=2713:00 +CPU= 21.414
  01:08:58 STEP 2714 H=2714:00 +CPU= 24.288
  01:09:41 STEP 2715 H=2715:00 +CPU= 34.263
  01:10:08 STEP 2716 H=2716:00 +CPU= 23.753
  01:10:32 STEP 2717 H=2717:00 +CPU= 20.781
  01:11:12 STEP 2718 H=2718:00 +CPU= 34.393
  01:11:41 STEP 2719 H=2719:00 +CPU= 23.113
  01:12:04 STEP 2720 H=2720:00 +CPU= 21.789
  01:12:45 STEP 2721 H=2721:00 +CPU= 32.524
  01:13:09 STEP 2722 H=2722:00 +CPU= 21.829
  01:13:33 STEP 2723 H=2723:00 +CPU= 21.761
  01:14:15 STEP 2724 H=2724:00 +CPU= 38.827
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002724
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002724
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002724
  01:14:43 STEP 2725 H=2725:00 +CPU= 23.225
  01:15:06 STEP 2726 H=2726:00 +CPU= 21.547
  01:15:46 STEP 2727 H=2727:00 +CPU= 35.085
  01:16:15 STEP 2728 H=2728:00 +CPU= 23.873
  01:16:42 STEP 2729 H=2729:00 +CPU= 21.621
  01:17:23 STEP 2730 H=2730:00 +CPU= 35.406
  01:17:48 STEP 2731 H=2731:00 +CPU= 21.919
  01:18:20 STEP 2732 H=2732:00 +CPU= 23.648
  01:19:01 STEP 2733 H=2733:00 +CPU= 34.645
  01:19:32 STEP 2734 H=2734:00 +CPU= 22.831
  01:19:58 STEP 2735 H=2735:00 +CPU= 23.089
  01:20:42 STEP 2736 H=2736:00 +CPU= 38.136
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002736
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002736
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002724
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002724
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002724
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_113.zip
Uploading the intermediate file: upload_file_113.zip
  01:21:06 STEP 2737 H=2737:00 +CPU= 20.830
  01:21:33 STEP 2738 H=2738:00 +CPU= 24.420
  01:22:12 STEP 2739 H=2739:00 +CPU= 34.854
  01:22:39 STEP 2740 H=2740:00 +CPU= 23.413
  01:23:04 STEP 2741 H=2741:00 +CPU= 22.498
  01:23:41 STEP 2742 H=2742:00 +CPU= 33.802
  01:24:05 STEP 2743 H=2743:00 +CPU= 20.479
  01:24:29 STEP 2744 H=2744:00 +CPU= 21.609
  01:25:04 STEP 2745 H=2745:00 +CPU= 31.216
  01:25:30 STEP 2746 H=2746:00 +CPU= 23.924
  01:25:56 STEP 2747 H=2747:00 +CPU= 21.297
  01:26:39 STEP 2748 H=2748:00 +CPU= 37.834
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002748
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002748
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002748
  01:27:05 STEP 2749 H=2749:00 +CPU= 21.645
  01:27:32 STEP 2750 H=2750:00 +CPU= 23.807
  01:28:10 STEP 2751 H=2751:00 +CPU= 35.503
  01:28:34 STEP 2752 H=2752:00 +CPU= 21.962
  01:29:02 STEP 2753 H=2753:00 +CPU= 22.228
  01:29:34 STEP 2754 H=2754:00 +CPU= 31.411
  01:29:58 STEP 2755 H=2755:00 +CPU= 21.245
  01:30:22 STEP 2756 H=2756:00 +CPU= 22.195
  01:31:05 STEP 2757 H=2757:00 +CPU= 34.430
  01:31:35 STEP 2758 H=2758:00 +CPU= 23.544
  01:32:00 STEP 2759 H=2759:00 +CPU= 22.940
  01:32:46 STEP 2760 H=2760:00 +CPU= 39.710
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002760
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002760
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002748
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002748
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002748
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_114.zip
Uploading the intermediate file: upload_file_114.zip
  01:33:10 STEP 2761 H=2761:00 +CPU= 21.722
  01:33:34 STEP 2762 H=2762:00 +CPU= 21.703
  01:34:14 STEP 2763 H=2763:00 +CPU= 33.463
  01:34:42 STEP 2764 H=2764:00 +CPU= 23.516
  01:35:10 STEP 2765 H=2765:00 +CPU= 22.766
  01:35:49 STEP 2766 H=2766:00 +CPU= 33.051
  01:36:14 STEP 2767 H=2767:00 +CPU= 22.102
  01:36:40 STEP 2768 H=2768:00 +CPU= 22.300
  01:37:18 STEP 2769 H=2769:00 +CPU= 32.755
  01:37:46 STEP 2770 H=2770:00 +CPU= 23.320
  01:38:11 STEP 2771 H=2771:00 +CPU= 22.021
  01:38:58 STEP 2772 H=2772:00 +CPU= 38.537
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002772
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002772
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002772
  01:39:24 STEP 2773 H=2773:00 +CPU= 22.281
  01:39:50 STEP 2774 H=2774:00 +CPU= 23.101
  01:40:25 STEP 2775 H=2775:00 +CPU= 32.121
  01:40:50 STEP 2776 H=2776:00 +CPU= 23.819
  01:41:14 STEP 2777 H=2777:00 +CPU= 22.303
  01:41:55 STEP 2778 H=2778:00 +CPU= 34.955
  01:42:20 STEP 2779 H=2779:00 +CPU= 22.477
  01:42:47 STEP 2780 H=2780:00 +CPU= 24.290
  01:43:24 STEP 2781 H=2781:00 +CPU= 32.787
  01:43:48 STEP 2782 H=2782:00 +CPU= 21.553
  01:44:15 STEP 2783 H=2783:00 +CPU= 23.835
  01:45:04 STEP 2784 H=2784:00 +CPU= 41.649
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002784
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002784
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002772
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002772
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002772
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_115.zip
Uploading the intermediate file: upload_file_115.zip
  01:45:30 STEP 2785 H=2785:00 +CPU= 22.888
  01:45:58 STEP 2786 H=2786:00 +CPU= 23.227
  01:46:31 STEP 2787 H=2787:00 +CPU= 31.435
  01:46:57 STEP 2788 H=2788:00 +CPU= 23.962
  01:47:25 STEP 2789 H=2789:00 +CPU= 22.569
  01:48:06 STEP 2790 H=2790:00 +CPU= 35.834
  01:48:30 STEP 2791 H=2791:00 +CPU= 21.101
  01:48:57 STEP 2792 H=2792:00 +CPU= 22.787
  01:49:37 STEP 2793 H=2793:00 +CPU= 35.428
  01:50:02 STEP 2794 H=2794:00 +CPU= 21.638
  01:50:26 STEP 2795 H=2795:00 +CPU= 22.101
  01:51:10 STEP 2796 H=2796:00 +CPU= 38.883
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002796
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002796
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002796
  01:51:36 STEP 2797 H=2797:00 +CPU= 23.029
  01:52:02 STEP 2798 H=2798:00 +CPU= 22.850
  01:52:44 STEP 2799 H=2799:00 +CPU= 35.293
  01:53:13 STEP 2800 H=2800:00 +CPU= 25.108
  01:53:36 STEP 2801 H=2801:00 +CPU= 20.652
  01:54:16 STEP 2802 H=2802:00 +CPU= 34.574
  01:54:40 STEP 2803 H=2803:00 +CPU= 22.242
  01:55:08 STEP 2804 H=2804:00 +CPU= 23.604
  01:55:47 STEP 2805 H=2805:00 +CPU= 33.746
  01:56:14 STEP 2806 H=2806:00 +CPU= 23.356
  01:56:42 STEP 2807 H=2807:00 +CPU= 22.563
  01:57:24 STEP 2808 H=2808:00 +CPU= 37.897
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002808
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002808
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002796
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002796
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002796
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_116.zip
Uploading the intermediate file: upload_file_116.zip
  01:57:50 STEP 2809 H=2809:00 +CPU= 23.522
  01:58:15 STEP 2810 H=2810:00 +CPU= 22.450
  01:58:53 STEP 2811 H=2811:00 +CPU= 34.564
  01:59:16 STEP 2812 H=2812:00 +CPU= 20.945
  01:59:38 STEP 2813 H=2813:00 +CPU= 19.752
  02:00:15 STEP 2814 H=2814:00 +CPU= 35.351
  02:00:39 STEP 2815 H=2815:00 +CPU= 23.107
  02:01:05 STEP 2816 H=2816:00 +CPU= 23.976
  02:01:47 STEP 2817 H=2817:00 +CPU= 34.604
  02:02:11 STEP 2818 H=2818:00 +CPU= 21.914
  02:02:35 STEP 2819 H=2819:00 +CPU= 21.836
  02:03:19 STEP 2820 H=2820:00 +CPU= 37.332
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002820
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002820
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002820
  02:03:42 STEP 2821 H=2821:00 +CPU= 20.231
  02:04:10 STEP 2822 H=2822:00 +CPU= 23.327
  02:04:55 STEP 2823 H=2823:00 +CPU= 35.667
  02:05:23 STEP 2824 H=2824:00 +CPU= 24.073
  02:05:45 STEP 2825 H=2825:00 +CPU= 20.904
  02:06:22 STEP 2826 H=2826:00 +CPU= 33.733
  02:06:49 STEP 2827 H=2827:00 +CPU= 23.243
  02:07:14 STEP 2828 H=2828:00 +CPU= 23.185
  02:07:50 STEP 2829 H=2829:00 +CPU= 34.355
  02:08:15 STEP 2830 H=2830:00 +CPU= 23.208
  02:08:42 STEP 2831 H=2831:00 +CPU= 24.417
  02:09:26 STEP 2832 H=2832:00 +CPU= 39.910
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002832
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002832
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002820
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002820
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002820
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_117.zip
Uploading the intermediate file: upload_file_117.zip
  02:09:53 STEP 2833 H=2833:00 +CPU= 21.418
  02:10:16 STEP 2834 H=2834:00 +CPU= 21.030
  02:10:53 STEP 2835 H=2835:00 +CPU= 34.760
  02:11:18 STEP 2836 H=2836:00 +CPU= 20.854
  02:11:45 STEP 2837 H=2837:00 +CPU= 24.159
  02:12:25 STEP 2838 H=2838:00 +CPU= 35.630
  02:12:50 STEP 2839 H=2839:00 +CPU= 22.418
  02:13:17 STEP 2840 H=2840:00 +CPU= 23.906
  02:13:54 STEP 2841 H=2841:00 +CPU= 33.939
  02:14:20 STEP 2842 H=2842:00 +CPU= 23.719
  02:14:45 STEP 2843 H=2843:00 +CPU= 22.258
  02:15:30 STEP 2844 H=2844:00 +CPU= 37.416
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002844
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002844
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002844
  02:15:56 STEP 2845 H=2845:00 +CPU= 22.487
  02:16:21 STEP 2846 H=2846:00 +CPU= 23.962
  02:17:03 STEP 2847 H=2847:00 +CPU= 36.044
  02:17:27 STEP 2848 H=2848:00 +CPU= 22.730
  02:17:50 STEP 2849 H=2849:00 +CPU= 21.093
  02:18:34 STEP 2850 H=2850:00 +CPU= 35.652
  02:18:56 STEP 2851 H=2851:00 +CPU= 20.592
  02:19:25 STEP 2852 H=2852:00 +CPU= 22.413
  02:20:07 STEP 2853 H=2853:00 +CPU= 34.089
  02:20:36 STEP 2854 H=2854:00 +CPU= 23.420
  02:21:03 STEP 2855 H=2855:00 +CPU= 22.488
  02:21:52 STEP 2856 H=2856:00 +CPU= 41.591
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002856
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002856
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002844
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002844
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002844
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_118.zip
Uploading the intermediate file: upload_file_118.zip
  02:22:18 STEP 2857 H=2857:00 +CPU= 22.645
  02:22:43 STEP 2858 H=2858:00 +CPU= 21.116
  02:23:17 STEP 2859 H=2859:00 +CPU= 31.210
  02:23:45 STEP 2860 H=2860:00 +CPU= 23.645
  02:24:13 STEP 2861 H=2861:00 +CPU= 23.527
  02:24:55 STEP 2862 H=2862:00 +CPU= 35.643
  02:25:21 STEP 2863 H=2863:00 +CPU= 22.204
  02:25:47 STEP 2864 H=2864:00 +CPU= 22.267
  02:26:24 STEP 2865 H=2865:00 +CPU= 32.520
  02:26:51 STEP 2866 H=2866:00 +CPU= 22.454
  02:27:17 STEP 2867 H=2867:00 +CPU= 21.963
  02:28:01 STEP 2868 H=2868:00 +CPU= 38.730
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002868
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002868
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002868
  02:28:27 STEP 2869 H=2869:00 +CPU= 22.388
  02:28:51 STEP 2870 H=2870:00 +CPU= 22.295
  02:29:29 STEP 2871 H=2871:00 +CPU= 33.583
  02:29:56 STEP 2872 H=2872:00 +CPU= 23.290
  02:30:17 STEP 2873 H=2873:00 +CPU= 20.176
  02:31:00 STEP 2874 H=2874:00 +CPU= 35.611
  02:31:26 STEP 2875 H=2875:00 +CPU= 22.251
  02:31:50 STEP 2876 H=2876:00 +CPU= 22.102
  02:32:29 STEP 2877 H=2877:00 +CPU= 35.025
  02:32:54 STEP 2878 H=2878:00 +CPU= 23.314
  02:33:22 STEP 2879 H=2879:00 +CPU= 24.018
  02:34:10 STEP 2880 H=2880:00 +CPU= 41.383
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002880
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002880
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002868
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002868
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002868
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_119.zip
Uploading the intermediate file: upload_file_119.zip
  02:34:37 STEP 2881 H=2881:00 +CPU= 23.023
Uploading trickle at timestep: 10368000
  02:35:02 STEP 2882 H=2882:00 +CPU= 22.824
  02:35:39 STEP 2883 H=2883:00 +CPU= 34.471
  02:36:09 STEP 2884 H=2884:00 +CPU= 22.923
  02:36:35 STEP 2885 H=2885:00 +CPU= 22.774
  02:37:13 STEP 2886 H=2886:00 +CPU= 33.110
  02:37:38 STEP 2887 H=2887:00 +CPU= 21.967
  02:38:06 STEP 2888 H=2888:00 +CPU= 23.854
  02:38:48 STEP 2889 H=2889:00 +CPU= 35.012
  02:39:09 STEP 2890 H=2890:00 +CPU= 20.078
  02:39:30 STEP 2891 H=2891:00 +CPU= 19.964
  02:40:14 STEP 2892 H=2892:00 +CPU= 37.772
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002892
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002892
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002892
  02:40:39 STEP 2893 H=2893:00 +CPU= 22.032
  02:41:06 STEP 2894 H=2894:00 +CPU= 23.451
  02:41:43 STEP 2895 H=2895:00 +CPU= 34.201
  02:42:07 STEP 2896 H=2896:00 +CPU= 22.261
  02:42:33 STEP 2897 H=2897:00 +CPU= 23.178
  02:43:11 STEP 2898 H=2898:00 +CPU= 33.568
  02:43:37 STEP 2899 H=2899:00 +CPU= 23.402
  02:44:01 STEP 2900 H=2900:00 +CPU= 22.334
  02:44:41 STEP 2901 H=2901:00 +CPU= 33.600
  02:45:05 STEP 2902 H=2902:00 +CPU= 21.506
  02:45:33 STEP 2903 H=2903:00 +CPU= 23.984
  02:46:23 STEP 2904 H=2904:00 +CPU= 41.850
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002904
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002904
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002892
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002892
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002892
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_120.zip
Uploading the intermediate file: upload_file_120.zip
  02:46:50 STEP 2905 H=2905:00 +CPU= 22.541
  02:47:16 STEP 2906 H=2906:00 +CPU= 23.253
  02:47:56 STEP 2907 H=2907:00 +CPU= 35.907
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1982050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0438
(argv4) batchid: 951
(argv5) wuid: 12168082
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/jeff/slots/21
Project directory is: /home/jeff/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082
..mkdir for temp folder for results failed
Copying: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/jeff/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/jeff/slots/21/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c45f313aeeec0b9e1897bf12709381ff to: /home/jeff/slots/21/oifs_43r3_ps_0438_1982050100_123_951_12168082.zip
Unzipping the namelist zip file: /home/jeff/slots/21/oifs_43r3_ps_0438_1982050100_123_951_12168082.zip
ic_ancil_file: ic_ancil_12168082
ifsdata_file: ifsdata_12168082
climate_data_file: clim_data_12168082
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_5c5f1b2a6ffc259b7b76a84c31453023 to: /home/jeff/slots/21/ic_ancil_12168082.zip
Unzipping the IC ancils zip file: /home/jeff/slots/21/ic_ancil_12168082.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a8a1772f80e6555063a90d87c5be2681 to: /home/jeff/slots/21/ifsdata/ifsdata_12168082.zip
Unzipping the ifsdata_zip file: /home/jeff/slots/21/ifsdata/ifsdata_12168082.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/jeff/slots/21/159l_2/clim_data_12168082.zip
Unzipping the climate data zip file: /home/jeff/slots/21/159l_2/clim_data_12168082.zip
Checking for progress XML file: /home/jeff/slots/21/progress_file_12168082.xml
Opened progress file ok : /home/jeff/slots/21/progress_file_12168082.xml
-- Model is restarting --
Adjusting last_iter, 2908, to previous model restart step.
Creating progress file: /home/jeff/slots/21/progress_file_12168082.xml
last_cpu_time: 82123
upload_file_number: 121
last_iter: 2903
last_upload: 10454400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178
The child process has been launched with process id: 10854
Executing the command: /home/jeff/slots/21/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at 0x1)
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10854:10854] [20230114:004349:1673678629:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:44:41 STEP 2904 H=2904:00 +CPU= 34.296
  00:45:05 STEP 2905 H=2905:00 +CPU= 20.676
  00:45:31 STEP 2906 H=2906:00 +CPU= 22.765
  00:46:14 STEP 2907 H=2907:00 +CPU= 34.834
  00:46:43 STEP 2908 H=2908:00 +CPU= 24.815
  00:47:08 STEP 2909 H=2909:00 +CPU= 22.503
  00:47:46 STEP 2910 H=2910:00 +CPU= 35.115
  00:48:14 STEP 2911 H=2911:00 +CPU= 24.528
  00:48:41 STEP 2912 H=2912:00 +CPU= 23.579
  00:49:19 STEP 2913 H=2913:00 +CPU= 33.868
  00:49:46 STEP 2914 H=2914:00 +CPU= 24.272
  00:50:12 STEP 2915 H=2915:00 +CPU= 23.361
  00:50:56 STEP 2916 H=2916:00 +CPU= 38.613
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002916
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002916
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002916
  00:51:22 STEP 2917 H=2917:00 +CPU= 21.774
  00:51:50 STEP 2918 H=2918:00 +CPU= 24.544
  00:52:34 STEP 2919 H=2919:00 +CPU= 36.273
  00:53:03 STEP 2920 H=2920:00 +CPU= 25.328
  00:53:30 STEP 2921 H=2921:00 +CPU= 23.641
  00:54:10 STEP 2922 H=2922:00 +CPU= 35.806
  00:54:36 STEP 2923 H=2923:00 +CPU= 23.767
  00:55:04 STEP 2924 H=2924:00 +CPU= 24.238
  00:55:44 STEP 2925 H=2925:00 +CPU= 36.325
  00:56:10 STEP 2926 H=2926:00 +CPU= 23.371
  00:56:37 STEP 2927 H=2927:00 +CPU= 24.519
  00:57:26 STEP 2928 H=2928:00 +CPU= 42.303
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002928
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002928
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002928
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002916
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002916
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002916
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_121.zip
Uploading the intermediate file: upload_file_121.zip
  00:57:52 STEP 2929 H=2929:00 +CPU= 22.987
  00:58:20 STEP 2930 H=2930:00 +CPU= 24.931
  00:59:03 STEP 2931 H=2931:00 +CPU= 35.780
  00:59:30 STEP 2932 H=2932:00 +CPU= 23.646
  00:59:55 STEP 2933 H=2933:00 +CPU= 22.002
  01:00:36 STEP 2934 H=2934:00 +CPU= 36.297
  01:01:03 STEP 2935 H=2935:00 +CPU= 24.380
  01:01:31 STEP 2936 H=2936:00 +CPU= 24.794
  01:02:14 STEP 2937 H=2937:00 +CPU= 36.878
  01:02:41 STEP 2938 H=2938:00 +CPU= 24.974
  01:03:10 STEP 2939 H=2939:00 +CPU= 24.160
  01:03:53 STEP 2940 H=2940:00 +CPU= 38.978
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002940
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002940
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002940
  01:04:21 STEP 2941 H=2941:00 +CPU= 23.946
  01:04:51 STEP 2942 H=2942:00 +CPU= 26.003
  01:05:32 STEP 2943 H=2943:00 +CPU= 36.179
  01:05:56 STEP 2944 H=2944:00 +CPU= 22.131
  01:06:25 STEP 2945 H=2945:00 +CPU= 23.911
  01:07:09 STEP 2946 H=2946:00 +CPU= 36.478
  01:07:34 STEP 2947 H=2947:00 +CPU= 23.018
  01:08:00 STEP 2948 H=2948:00 +CPU= 23.097
  01:08:41 STEP 2949 H=2949:00 +CPU= 35.589
  01:09:07 STEP 2950 H=2950:00 +CPU= 23.613
  01:09:37 STEP 2951 H=2951:00 +CPU= 25.281
  01:10:24 STEP 2952 H=2952:00 +CPU= 40.244
The child process terminated with status: 0
Moving to projects directory: /home/jeff/slots/21/ICMGGhq0f+002952
Moving to projects directory: /home/jeff/slots/21/ICMSHhq0f+002952
Moving to projects directory: /home/jeff/slots/21/ICMUAhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002928
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002928
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002940
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002940
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMSHhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMGGhq0f+002940
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12168082/ICMUAhq0f+002928
Zipping up the final file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0438_1982050100_123_951_12168082_0_r1519635178_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
01:13:47 (10819): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:18:55 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 10,623,600 83,510 0.0079
09 May 2023 16:17:50 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 10,368,000 81,388 0.0078
09 May 2023 16:17:43 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 9,504,000 74,959 0.0079
09 May 2023 16:17:37 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 8,640,000 68,510 0.0079
09 May 2023 16:17:29 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 7,776,000 61,878 0.0080
09 May 2023 16:17:20 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 6,912,000 54,779 0.0079
09 May 2023 16:17:10 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 6,048,000 47,527 0.0079
09 May 2023 16:17:03 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 5,184,000 40,366 0.0078
09 May 2023 16:16:52 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 4,320,000 33,452 0.0077
09 May 2023 16:16:42 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 3,456,000 26,452 0.0077
09 May 2023 16:16:33 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 2,592,000 19,728 0.0076
09 May 2023 16:16:23 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 1,728,000 13,284 0.0077
09 May 2023 16:16:14 1538024 22252323 oifs_43r3_ps_0438_1982050100_123_951_12168082_0 864,000 6,808 0.0079


©2024 cpdn.org