climateprediction.net home page
Task 22292792

Task 22292792

Name oifs_43r3_ps_0558_2020050100_123_989_12206202_0
Workunit 12206202
Created 25 Dec 2022, 2:17:44 UTC
Sent 12 Jan 2023, 0:18:29 UTC
Report deadline 11 Feb 2023, 0:18:29 UTC
Received 14 Jan 2023, 9:13:17 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1545600
Run time 1 days 8 hours 59 min 10 sec
CPU time 23 hours 20 min 45 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 6.17 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,411.05 MB
Peak swap size 4,978.08 MB
Peak disk usage 3,307.51 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
=2390:00 +CPU= 20.146
  22:23:37 STEP 2391 H=2391:00 +CPU= 33.633
  22:24:03 STEP 2392 H=2392:00 +CPU= 23.453
  22:24:28 STEP 2393 H=2393:00 +CPU= 23.820
  22:25:10 STEP 2394 H=2394:00 +CPU= 35.088
  22:25:34 STEP 2395 H=2395:00 +CPU= 22.125
  22:26:03 STEP 2396 H=2396:00 +CPU= 23.609
  22:26:48 STEP 2397 H=2397:00 +CPU= 35.849
  22:27:14 STEP 2398 H=2398:00 +CPU= 22.950
  22:27:42 STEP 2399 H=2399:00 +CPU= 24.272
  22:28:31 STEP 2400 H=2400:00 +CPU= 42.049
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002400
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002400
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002376
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002376
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002376
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002388
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002388
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002388
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_99.zip
Uploading the intermediate file: upload_file_99.zip
  22:28:56 STEP 2401 H=2401:00 +CPU= 22.919
Uploading trickle at timestep: 8640000
  22:29:19 STEP 2402 H=2402:00 +CPU= 21.284
  22:30:02 STEP 2403 H=2403:00 +CPU= 35.196
  22:30:28 STEP 2404 H=2404:00 +CPU= 22.562
  22:30:58 STEP 2405 H=2405:00 +CPU= 23.886
  22:31:42 STEP 2406 H=2406:00 +CPU= 35.960
  22:32:10 STEP 2407 H=2407:00 +CPU= 23.257
  22:32:37 STEP 2408 H=2408:00 +CPU= 22.955
  22:33:18 STEP 2409 H=2409:00 +CPU= 33.661
  22:33:48 STEP 2410 H=2410:00 +CPU= 25.840
  22:34:17 STEP 2411 H=2411:00 +CPU= 23.762
  22:35:05 STEP 2412 H=2412:00 +CPU= 38.321
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002412
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002412
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002412
  22:35:36 STEP 2413 H=2413:00 +CPU= 23.450
  22:36:04 STEP 2414 H=2414:00 +CPU= 25.184
  22:36:42 STEP 2415 H=2415:00 +CPU= 32.839
  22:37:09 STEP 2416 H=2416:00 +CPU= 23.419
  22:37:31 STEP 2417 H=2417:00 +CPU= 19.774
  22:38:12 STEP 2418 H=2418:00 +CPU= 35.622
  22:38:41 STEP 2419 H=2419:00 +CPU= 23.196
  22:39:04 STEP 2420 H=2420:00 +CPU= 21.970
  22:39:40 STEP 2421 H=2421:00 +CPU= 31.327
  22:40:06 STEP 2422 H=2422:00 +CPU= 22.497
  22:40:34 STEP 2423 H=2423:00 +CPU= 23.036
  22:41:23 STEP 2424 H=2424:00 +CPU= 41.288
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002424
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002424
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002400
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002412
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002412
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002412
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_100.zip
Uploading the intermediate file: upload_file_100.zip
  22:41:47 STEP 2425 H=2425:00 +CPU= 22.178
  22:42:13 STEP 2426 H=2426:00 +CPU= 22.962
  22:42:54 STEP 2427 H=2427:00 +CPU= 34.341
  22:43:22 STEP 2428 H=2428:00 +CPU= 23.736
  22:43:48 STEP 2429 H=2429:00 +CPU= 23.016
  22:44:29 STEP 2430 H=2430:00 +CPU= 37.144
  22:44:58 STEP 2431 H=2431:00 +CPU= 24.267
  22:45:23 STEP 2432 H=2432:00 +CPU= 22.538
  22:46:05 STEP 2433 H=2433:00 +CPU= 35.127
  22:46:29 STEP 2434 H=2434:00 +CPU= 21.753
  22:46:55 STEP 2435 H=2435:00 +CPU= 23.739
  22:47:42 STEP 2436 H=2436:00 +CPU= 38.710
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002436
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002436
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002436
  22:48:07 STEP 2437 H=2437:00 +CPU= 22.141
  22:48:35 STEP 2438 H=2438:00 +CPU= 24.008
  22:49:19 STEP 2439 H=2439:00 +CPU= 35.802
  22:49:46 STEP 2440 H=2440:00 +CPU= 23.768
  22:50:12 STEP 2441 H=2441:00 +CPU= 21.918
  22:50:54 STEP 2442 H=2442:00 +CPU= 36.555
  22:51:19 STEP 2443 H=2443:00 +CPU= 22.331
  22:51:45 STEP 2444 H=2444:00 +CPU= 22.572
  22:52:27 STEP 2445 H=2445:00 +CPU= 34.808
  22:52:54 STEP 2446 H=2446:00 +CPU= 23.894
  22:53:20 STEP 2447 H=2447:00 +CPU= 22.998
  22:54:06 STEP 2448 H=2448:00 +CPU= 40.742
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002448
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002448
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002424
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002436
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002436
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002436
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_101.zip
Uploading the intermediate file: upload_file_101.zip
  22:54:32 STEP 2449 H=2449:00 +CPU= 22.352
  22:54:56 STEP 2450 H=2450:00 +CPU= 22.591
  22:55:39 STEP 2451 H=2451:00 +CPU= 36.340
  22:56:06 STEP 2452 H=2452:00 +CPU= 23.301
  22:56:31 STEP 2453 H=2453:00 +CPU= 22.972
  22:57:08 STEP 2454 H=2454:00 +CPU= 34.830
  22:57:34 STEP 2455 H=2455:00 +CPU= 22.625
  22:57:58 STEP 2456 H=2456:00 +CPU= 22.447
  22:58:37 STEP 2457 H=2457:00 +CPU= 34.027
  22:59:03 STEP 2458 H=2458:00 +CPU= 22.924
  22:59:24 STEP 2459 H=2459:00 +CPU= 19.183
  23:00:05 STEP 2460 H=2460:00 +CPU= 36.515
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002460
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002460
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002460
  23:00:28 STEP 2461 H=2461:00 +CPU= 21.602
  23:00:55 STEP 2462 H=2462:00 +CPU= 22.507
  23:01:34 STEP 2463 H=2463:00 +CPU= 35.681
  23:01:58 STEP 2464 H=2464:00 +CPU= 21.585
  23:02:26 STEP 2465 H=2465:00 +CPU= 24.021
  23:03:05 STEP 2466 H=2466:00 +CPU= 34.575
  23:03:30 STEP 2467 H=2467:00 +CPU= 21.848
  23:03:57 STEP 2468 H=2468:00 +CPU= 23.357
  23:04:38 STEP 2469 H=2469:00 +CPU= 34.406
  23:05:05 STEP 2470 H=2470:00 +CPU= 23.289
  23:05:30 STEP 2471 H=2471:00 +CPU= 22.238
  23:06:13 STEP 2472 H=2472:00 +CPU= 39.615
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002472
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002472
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002448
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002460
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002460
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002460
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_102.zip
Uploading the intermediate file: upload_file_102.zip
  23:06:40 STEP 2473 H=2473:00 +CPU= 24.217
  23:07:05 STEP 2474 H=2474:00 +CPU= 22.872
  23:07:49 STEP 2475 H=2475:00 +CPU= 36.331
  23:08:19 STEP 2476 H=2476:00 +CPU= 24.417
  23:08:50 STEP 2477 H=2477:00 +CPU= 24.552
  23:09:28 STEP 2478 H=2478:00 +CPU= 35.206
  23:09:53 STEP 2479 H=2479:00 +CPU= 21.387
  23:10:18 STEP 2480 H=2480:00 +CPU= 23.581
  23:10:57 STEP 2481 H=2481:00 +CPU= 34.789
  23:11:22 STEP 2482 H=2482:00 +CPU= 22.202
  23:11:44 STEP 2483 H=2483:00 +CPU= 20.628
  23:12:28 STEP 2484 H=2484:00 +CPU= 36.845
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002484
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002484
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002484
  23:12:51 STEP 2485 H=2485:00 +CPU= 21.136
  23:13:15 STEP 2486 H=2486:00 +CPU= 21.734
  23:13:53 STEP 2487 H=2487:00 +CPU= 33.895
  23:14:18 STEP 2488 H=2488:00 +CPU= 22.723
  23:14:41 STEP 2489 H=2489:00 +CPU= 20.683
  23:15:23 STEP 2490 H=2490:00 +CPU= 37.111
  23:15:49 STEP 2491 H=2491:00 +CPU= 23.945
  23:16:14 STEP 2492 H=2492:00 +CPU= 23.300
  23:16:53 STEP 2493 H=2493:00 +CPU= 34.673
  23:17:21 STEP 2494 H=2494:00 +CPU= 22.491
  23:17:48 STEP 2495 H=2495:00 +CPU= 22.699
  23:18:34 STEP 2496 H=2496:00 +CPU= 38.242
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002496
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002496
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002472
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002484
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002484
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002484
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_103.zip
Uploading the intermediate file: upload_file_103.zip
  23:18:56 STEP 2497 H=2497:00 +CPU= 21.384
  23:19:21 STEP 2498 H=2498:00 +CPU= 22.682
  23:20:00 STEP 2499 H=2499:00 +CPU= 33.847
  23:20:25 STEP 2500 H=2500:00 +CPU= 22.565
  23:20:51 STEP 2501 H=2501:00 +CPU= 22.096
  23:21:36 STEP 2502 H=2502:00 +CPU= 36.698
  23:22:00 STEP 2503 H=2503:00 +CPU= 21.974
  23:22:24 STEP 2504 H=2504:00 +CPU= 21.652
  23:23:07 STEP 2505 H=2505:00 +CPU= 34.960
  23:23:34 STEP 2506 H=2506:00 +CPU= 24.120
  23:24:04 STEP 2507 H=2507:00 +CPU= 22.693
  23:24:47 STEP 2508 H=2508:00 +CPU= 36.816
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002508
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002508
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002508
  23:25:12 STEP 2509 H=2509:00 +CPU= 22.577
  23:25:36 STEP 2510 H=2510:00 +CPU= 21.539
  23:26:16 STEP 2511 H=2511:00 +CPU= 34.140
  23:26:42 STEP 2512 H=2512:00 +CPU= 22.522
  23:27:09 STEP 2513 H=2513:00 +CPU= 21.782
  23:27:48 STEP 2514 H=2514:00 +CPU= 34.613
  23:28:12 STEP 2515 H=2515:00 +CPU= 20.121
  23:28:37 STEP 2516 H=2516:00 +CPU= 20.969
  23:29:21 STEP 2517 H=2517:00 +CPU= 34.532
  23:29:45 STEP 2518 H=2518:00 +CPU= 21.316
  23:30:15 STEP 2519 H=2519:00 +CPU= 23.028
  23:30:59 STEP 2520 H=2520:00 +CPU= 41.832
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002520
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002520
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002496
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002508
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002508
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002508
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_104.zip
Uploading the intermediate file: upload_file_104.zip
  23:31:26 STEP 2521 H=2521:00 +CPU= 21.672
  23:31:51 STEP 2522 H=2522:00 +CPU= 22.440
  23:32:26 STEP 2523 H=2523:00 +CPU= 32.247
  23:32:53 STEP 2524 H=2524:00 +CPU= 22.994
  23:33:19 STEP 2525 H=2525:00 +CPU= 22.868
  23:34:00 STEP 2526 H=2526:00 +CPU= 35.154
  23:34:25 STEP 2527 H=2527:00 +CPU= 23.237
  23:34:51 STEP 2528 H=2528:00 +CPU= 22.473
  23:35:32 STEP 2529 H=2529:00 +CPU= 35.464
  23:35:59 STEP 2530 H=2530:00 +CPU= 23.003
  23:36:25 STEP 2531 H=2531:00 +CPU= 23.223
  23:37:10 STEP 2532 H=2532:00 +CPU= 39.162
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002532
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002532
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002532
  23:37:36 STEP 2533 H=2533:00 +CPU= 23.970
  23:38:01 STEP 2534 H=2534:00 +CPU= 22.945
  23:38:42 STEP 2535 H=2535:00 +CPU= 34.808
  23:39:09 STEP 2536 H=2536:00 +CPU= 23.151
  23:39:35 STEP 2537 H=2537:00 +CPU= 23.406
  23:40:11 STEP 2538 H=2538:00 +CPU= 33.292
  23:40:36 STEP 2539 H=2539:00 +CPU= 21.644
  23:41:04 STEP 2540 H=2540:00 +CPU= 24.345
  23:41:41 STEP 2541 H=2541:00 +CPU= 33.799
  23:42:05 STEP 2542 H=2542:00 +CPU= 21.036
  23:42:30 STEP 2543 H=2543:00 +CPU= 22.121
  23:43:11 STEP 2544 H=2544:00 +CPU= 37.759
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002544
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002544
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002520
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002532
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002532
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002532
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_105.zip
Uploading the intermediate file: upload_file_105.zip
  23:43:37 STEP 2545 H=2545:00 +CPU= 23.747
  23:44:03 STEP 2546 H=2546:00 +CPU= 24.095
  23:44:40 STEP 2547 H=2547:00 +CPU= 34.307
  23:45:04 STEP 2548 H=2548:00 +CPU= 21.366
  23:45:29 STEP 2549 H=2549:00 +CPU= 23.675
  23:46:08 STEP 2550 H=2550:00 +CPU= 36.056
  23:46:36 STEP 2551 H=2551:00 +CPU= 23.346
  23:47:04 STEP 2552 H=2552:00 +CPU= 24.388
  23:47:43 STEP 2553 H=2553:00 +CPU= 34.530
  23:48:07 STEP 2554 H=2554:00 +CPU= 21.776
  23:48:36 STEP 2555 H=2555:00 +CPU= 24.710
  23:49:19 STEP 2556 H=2556:00 +CPU= 37.144
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002556
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002556
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002556
  23:49:46 STEP 2557 H=2557:00 +CPU= 23.529
  23:50:11 STEP 2558 H=2558:00 +CPU= 22.255
  23:50:52 STEP 2559 H=2559:00 +CPU= 34.725
  23:51:15 STEP 2560 H=2560:00 +CPU= 20.498
  23:51:40 STEP 2561 H=2561:00 +CPU= 21.635
  23:52:15 STEP 2562 H=2562:00 +CPU= 33.111
  23:52:40 STEP 2563 H=2563:00 +CPU= 22.424
  23:53:05 STEP 2564 H=2564:00 +CPU= 22.154
  23:53:43 STEP 2565 H=2565:00 +CPU= 34.170
  23:54:08 STEP 2566 H=2566:00 +CPU= 22.724
  23:54:34 STEP 2567 H=2567:00 +CPU= 22.051
  23:55:20 STEP 2568 H=2568:00 +CPU= 40.107
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002568
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002568
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002544
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002556
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002556
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002556
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_106.zip
Uploading the intermediate file: upload_file_106.zip
  23:55:44 STEP 2569 H=2569:00 +CPU= 20.973
  23:56:10 STEP 2570 H=2570:00 +CPU= 23.421
  23:56:53 STEP 2571 H=2571:00 +CPU= 36.970
  23:57:19 STEP 2572 H=2572:00 +CPU= 22.629
  23:57:42 STEP 2573 H=2573:00 +CPU= 21.643
  23:58:18 STEP 2574 H=2574:00 +CPU= 33.168
  23:58:45 STEP 2575 H=2575:00 +CPU= 22.859
  23:59:14 STEP 2576 H=2576:00 +CPU= 23.680
  23:59:52 STEP 2577 H=2577:00 +CPU= 33.604
  00:00:16 STEP 2578 H=2578:00 +CPU= 20.846
  00:00:38 STEP 2579 H=2579:00 +CPU= 20.777
  00:01:19 STEP 2580 H=2580:00 +CPU= 36.823
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002580
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002580
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002580
  00:01:44 STEP 2581 H=2581:00 +CPU= 21.027
  00:02:12 STEP 2582 H=2582:00 +CPU= 23.499
  00:02:53 STEP 2583 H=2583:00 +CPU= 35.192
  00:03:20 STEP 2584 H=2584:00 +CPU= 23.739
  00:03:42 STEP 2585 H=2585:00 +CPU= 20.236
  00:04:22 STEP 2586 H=2586:00 +CPU= 35.566
  00:04:48 STEP 2587 H=2587:00 +CPU= 22.602
  00:05:13 STEP 2588 H=2588:00 +CPU= 23.121
  00:05:53 STEP 2589 H=2589:00 +CPU= 33.515
  00:06:19 STEP 2590 H=2590:00 +CPU= 23.072
  00:06:47 STEP 2591 H=2591:00 +CPU= 23.324
  00:07:28 STEP 2592 H=2592:00 +CPU= 38.752
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002592
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002592
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002568
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002580
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002580
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002580
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_107.zip
Uploading the intermediate file: upload_file_107.zip
  00:07:52 STEP 2593 H=2593:00 +CPU= 20.646
  00:08:18 STEP 2594 H=2594:00 +CPU= 24.015
  00:08:55 STEP 2595 H=2595:00 +CPU= 33.183
  00:09:21 STEP 2596 H=2596:00 +CPU= 21.959
  00:09:45 STEP 2597 H=2597:00 +CPU= 21.073
  00:10:23 STEP 2598 H=2598:00 +CPU= 33.523
  00:10:49 STEP 2599 H=2599:00 +CPU= 21.690
  00:11:14 STEP 2600 H=2600:00 +CPU= 21.753
  00:11:53 STEP 2601 H=2601:00 +CPU= 35.013
  00:12:18 STEP 2602 H=2602:00 +CPU= 22.472
  00:12:42 STEP 2603 H=2603:00 +CPU= 21.398
  00:13:21 STEP 2604 H=2604:00 +CPU= 35.336
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002604
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002604
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002604
  00:13:46 STEP 2605 H=2605:00 +CPU= 21.429
  00:14:09 STEP 2606 H=2606:00 +CPU= 22.133
  00:14:47 STEP 2607 H=2607:00 +CPU= 33.122
  00:15:14 STEP 2608 H=2608:00 +CPU= 23.076
  00:15:41 STEP 2609 H=2609:00 +CPU= 22.992
  00:16:26 STEP 2610 H=2610:00 +CPU= 35.425
  00:16:50 STEP 2611 H=2611:00 +CPU= 21.214
  00:17:12 STEP 2612 H=2612:00 +CPU= 20.350
  00:17:45 STEP 2613 H=2613:00 +CPU= 32.279
  00:18:12 STEP 2614 H=2614:00 +CPU= 22.282
  00:18:39 STEP 2615 H=2615:00 +CPU= 22.340
  00:19:28 STEP 2616 H=2616:00 +CPU= 40.841
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002616
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002616
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002592
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002604
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002604
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002604
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_108.zip
Uploading the intermediate file: upload_file_108.zip
  00:19:53 STEP 2617 H=2617:00 +CPU= 21.187
  00:20:16 STEP 2618 H=2618:00 +CPU= 20.644
  00:20:56 STEP 2619 H=2619:00 +CPU= 34.295
  00:21:25 STEP 2620 H=2620:00 +CPU= 22.979
  00:21:51 STEP 2621 H=2621:00 +CPU= 22.156
  00:22:27 STEP 2622 H=2622:00 +CPU= 32.635
  00:22:47 STEP 2623 H=2623:00 +CPU= 19.473
  00:23:15 STEP 2624 H=2624:00 +CPU= 22.223
  00:23:52 STEP 2625 H=2625:00 +CPU= 32.468
  00:24:16 STEP 2626 H=2626:00 +CPU= 21.772
  00:24:39 STEP 2627 H=2627:00 +CPU= 21.470
  00:25:19 STEP 2628 H=2628:00 +CPU= 36.567
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002628
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002628
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002628
  00:25:45 STEP 2629 H=2629:00 +CPU= 23.230
  00:26:10 STEP 2630 H=2630:00 +CPU= 22.348
  00:26:50 STEP 2631 H=2631:00 +CPU= 33.781
  00:27:15 STEP 2632 H=2632:00 +CPU= 22.416
  00:27:38 STEP 2633 H=2633:00 +CPU= 21.127
  00:28:21 STEP 2634 H=2634:00 +CPU= 33.663
  00:28:48 STEP 2635 H=2635:00 +CPU= 22.294
  00:29:19 STEP 2636 H=2636:00 +CPU= 23.107
  00:30:02 STEP 2637 H=2637:00 +CPU= 34.715
  00:30:26 STEP 2638 H=2638:00 +CPU= 22.008
  00:30:52 STEP 2639 H=2639:00 +CPU= 22.453
  00:31:38 STEP 2640 H=2640:00 +CPU= 39.848
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002640
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002640
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002616
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002628
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002628
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002628
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_109.zip
Uploading the intermediate file: upload_file_109.zip
  00:32:01 STEP 2641 H=2641:00 +CPU= 21.269
Uploading trickle at timestep: 9504000
  00:32:23 STEP 2642 H=2642:00 +CPU= 20.392
  00:32:55 STEP 2643 H=2643:00 +CPU= 30.768
  00:33:19 STEP 2644 H=2644:00 +CPU= 21.362
  00:33:52 STEP 2645 H=2645:00 +CPU= 23.762
  00:34:35 STEP 2646 H=2646:00 +CPU= 35.825
  00:34:56 STEP 2647 H=2647:00 +CPU= 19.410
  00:35:20 STEP 2648 H=2648:00 +CPU= 20.878
  00:35:54 STEP 2649 H=2649:00 +CPU= 31.693
  00:36:19 STEP 2650 H=2650:00 +CPU= 24.488
  00:36:50 STEP 2651 H=2651:00 +CPU= 23.951
  00:37:34 STEP 2652 H=2652:00 +CPU= 37.470
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002652
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002652
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002652
  00:37:56 STEP 2653 H=2653:00 +CPU= 20.167
  00:38:21 STEP 2654 H=2654:00 +CPU= 22.577
  00:39:02 STEP 2655 H=2655:00 +CPU= 34.384
  00:39:30 STEP 2656 H=2656:00 +CPU= 23.250
  00:39:58 STEP 2657 H=2657:00 +CPU= 22.692
  00:40:39 STEP 2658 H=2658:00 +CPU= 35.644
  00:41:07 STEP 2659 H=2659:00 +CPU= 22.336
  00:41:34 STEP 2660 H=2660:00 +CPU= 22.871
  00:42:10 STEP 2661 H=2661:00 +CPU= 33.649
  00:42:35 STEP 2662 H=2662:00 +CPU= 20.969
  00:43:00 STEP 2663 H=2663:00 +CPU= 21.680
  00:43:48 STEP 2664 H=2664:00 +CPU= 40.329
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002664
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002664
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002640
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002652
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002652
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002652
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_110.zip
Uploading the intermediate file: upload_file_110.zip
  00:44:16 STEP 2665 H=2665:00 +CPU= 22.847
  00:44:48 STEP 2666 H=2666:00 +CPU= 23.469
  00:45:30 STEP 2667 H=2667:00 +CPU= 35.087
  00:45:56 STEP 2668 H=2668:00 +CPU= 22.603
  00:46:23 STEP 2669 H=2669:00 +CPU= 21.700
  00:47:10 STEP 2670 H=2670:00 +CPU= 36.122
  00:47:31 STEP 2671 H=2671:00 +CPU= 20.100
  00:47:58 STEP 2672 H=2672:00 +CPU= 21.702
  00:48:34 STEP 2673 H=2673:00 +CPU= 31.873
  00:49:00 STEP 2674 H=2674:00 +CPU= 22.426
  00:49:26 STEP 2675 H=2675:00 +CPU= 22.357
  00:50:07 STEP 2676 H=2676:00 +CPU= 37.057
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002676
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002676
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002676
  00:50:35 STEP 2677 H=2677:00 +CPU= 22.198
  00:50:59 STEP 2678 H=2678:00 +CPU= 21.421
  00:51:37 STEP 2679 H=2679:00 +CPU= 35.079
  00:52:04 STEP 2680 H=2680:00 +CPU= 24.058
  00:52:35 STEP 2681 H=2681:00 +CPU= 23.025
  00:53:19 STEP 2682 H=2682:00 +CPU= 35.576
  00:53:45 STEP 2683 H=2683:00 +CPU= 21.223
  00:54:12 STEP 2684 H=2684:00 +CPU= 22.749
  00:54:49 STEP 2685 H=2685:00 +CPU= 32.502
  00:55:16 STEP 2686 H=2686:00 +CPU= 21.431
  00:55:46 STEP 2687 H=2687:00 +CPU= 22.930
  00:56:36 STEP 2688 H=2688:00 +CPU= 41.672
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002688
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002688
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002664
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002676
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002676
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002676
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_111.zip
Uploading the intermediate file: upload_file_111.zip
  00:57:02 STEP 2689 H=2689:00 +CPU= 21.718
  00:57:34 STEP 2690 H=2690:00 +CPU= 23.573
  00:58:14 STEP 2691 H=2691:00 +CPU= 34.291
  00:58:42 STEP 2692 H=2692:00 +CPU= 23.717
  00:59:04 STEP 2693 H=2693:00 +CPU= 20.746
  00:59:45 STEP 2694 H=2694:00 +CPU= 35.957
  01:00:09 STEP 2695 H=2695:00 +CPU= 21.307
  01:00:36 STEP 2696 H=2696:00 +CPU= 22.509
  01:01:17 STEP 2697 H=2697:00 +CPU= 34.770
  01:01:39 STEP 2698 H=2698:00 +CPU= 22.418
  01:02:04 STEP 2699 H=2699:00 +CPU= 22.925
  01:02:51 STEP 2700 H=2700:00 +CPU= 37.506
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002700
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002700
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002700
  01:03:17 STEP 2701 H=2701:00 +CPU= 22.189
  01:03:43 STEP 2702 H=2702:00 +CPU= 22.977
  01:04:25 STEP 2703 H=2703:00 +CPU= 35.234
  01:04:52 STEP 2704 H=2704:00 +CPU= 24.743
  01:05:22 STEP 2705 H=2705:00 +CPU= 23.142
  01:06:04 STEP 2706 H=2706:00 +CPU= 35.548
  01:06:31 STEP 2707 H=2707:00 +CPU= 22.900
  01:06:57 STEP 2708 H=2708:00 +CPU= 22.037
  01:07:33 STEP 2709 H=2709:00 +CPU= 31.081
  01:07:59 STEP 2710 H=2710:00 +CPU= 20.956
  01:08:28 STEP 2711 H=2711:00 +CPU= 23.604
  01:09:18 STEP 2712 H=2712:00 +CPU= 42.035
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002712
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002712
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002688
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002700
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002700
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002700
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_112.zip
Uploading the intermediate file: upload_file_112.zip
  01:09:46 STEP 2713 H=2713:00 +CPU= 22.396
  01:10:10 STEP 2714 H=2714:00 +CPU= 22.704
  01:10:50 STEP 2715 H=2715:00 +CPU= 32.541
  01:11:20 STEP 2716 H=2716:00 +CPU= 22.561
  01:11:44 STEP 2717 H=2717:00 +CPU= 22.262
  01:12:24 STEP 2718 H=2718:00 +CPU= 34.562
  01:12:49 STEP 2719 H=2719:00 +CPU= 20.579
  01:13:12 STEP 2720 H=2720:00 +CPU= 22.205
  01:13:52 STEP 2721 H=2721:00 +CPU= 33.781
  01:14:20 STEP 2722 H=2722:00 +CPU= 22.766
  01:14:45 STEP 2723 H=2723:00 +CPU= 22.852
  01:15:26 STEP 2724 H=2724:00 +CPU= 37.245
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002724
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002724
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002724
  01:15:53 STEP 2725 H=2725:00 +CPU= 22.901
  01:16:18 STEP 2726 H=2726:00 +CPU= 22.503
  01:16:59 STEP 2727 H=2727:00 +CPU= 34.980
  01:17:26 STEP 2728 H=2728:00 +CPU= 22.800
  01:17:56 STEP 2729 H=2729:00 +CPU= 23.082
  01:18:40 STEP 2730 H=2730:00 +CPU= 35.777
  01:19:05 STEP 2731 H=2731:00 +CPU= 22.017
  01:19:30 STEP 2732 H=2732:00 +CPU= 21.307
  01:20:13 STEP 2733 H=2733:00 +CPU= 33.553
  01:20:37 STEP 2734 H=2734:00 +CPU= 22.393
  01:21:02 STEP 2735 H=2735:00 +CPU= 22.664
  01:21:46 STEP 2736 H=2736:00 +CPU= 39.056
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002736
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002736
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002712
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002724
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002724
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002724
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_113.zip
Uploading the intermediate file: upload_file_113.zip
  01:22:17 STEP 2737 H=2737:00 +CPU= 24.061
  01:22:45 STEP 2738 H=2738:00 +CPU= 23.174
  01:23:28 STEP 2739 H=2739:00 +CPU= 35.965
  01:23:57 STEP 2740 H=2740:00 +CPU= 23.631
  01:24:25 STEP 2741 H=2741:00 +CPU= 22.873
  01:25:07 STEP 2742 H=2742:00 +CPU= 35.448
  01:25:31 STEP 2743 H=2743:00 +CPU= 21.635
  01:25:58 STEP 2744 H=2744:00 +CPU= 23.390
  01:26:37 STEP 2745 H=2745:00 +CPU= 35.273
  01:27:04 STEP 2746 H=2746:00 +CPU= 23.395
  01:27:30 STEP 2747 H=2747:00 +CPU= 23.300
  01:28:13 STEP 2748 H=2748:00 +CPU= 36.679
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002748
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002748
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002748
  01:28:37 STEP 2749 H=2749:00 +CPU= 20.979
  01:29:04 STEP 2750 H=2750:00 +CPU= 22.855
  01:29:47 STEP 2751 H=2751:00 +CPU= 34.838
  01:30:10 STEP 2752 H=2752:00 +CPU= 20.616
  01:30:33 STEP 2753 H=2753:00 +CPU= 20.645
  01:31:11 STEP 2754 H=2754:00 +CPU= 33.168
  01:31:33 STEP 2755 H=2755:00 +CPU= 20.634
  01:31:57 STEP 2756 H=2756:00 +CPU= 21.808
  01:32:33 STEP 2757 H=2757:00 +CPU= 32.991
  01:32:58 STEP 2758 H=2758:00 +CPU= 22.012
  01:33:23 STEP 2759 H=2759:00 +CPU= 22.607
  01:34:12 STEP 2760 H=2760:00 +CPU= 40.065
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002760
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002760
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002736
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002748
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002748
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002748
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_114.zip
Uploading the intermediate file: upload_file_114.zip
  01:34:39 STEP 2761 H=2761:00 +CPU= 22.414
  01:35:04 STEP 2762 H=2762:00 +CPU= 22.492
  01:35:43 STEP 2763 H=2763:00 +CPU= 33.360
  01:36:08 STEP 2764 H=2764:00 +CPU= 22.085
  01:36:35 STEP 2765 H=2765:00 +CPU= 21.939
  01:37:13 STEP 2766 H=2766:00 +CPU= 34.266
  01:37:41 STEP 2767 H=2767:00 +CPU= 23.078
  01:38:10 STEP 2768 H=2768:00 +CPU= 24.341
  01:38:49 STEP 2769 H=2769:00 +CPU= 33.476
  01:39:15 STEP 2770 H=2770:00 +CPU= 23.439
  01:39:40 STEP 2771 H=2771:00 +CPU= 21.142
  01:40:25 STEP 2772 H=2772:00 +CPU= 39.083
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002772
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002772
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002772
  01:40:53 STEP 2773 H=2773:00 +CPU= 22.525
  01:41:20 STEP 2774 H=2774:00 +CPU= 23.057
  01:41:58 STEP 2775 H=2775:00 +CPU= 35.617
  01:42:26 STEP 2776 H=2776:00 +CPU= 24.249
  01:42:51 STEP 2777 H=2777:00 +CPU= 23.348
  01:43:36 STEP 2778 H=2778:00 +CPU= 36.223
  01:44:04 STEP 2779 H=2779:00 +CPU= 23.251
  01:44:28 STEP 2780 H=2780:00 +CPU= 22.408
  01:45:09 STEP 2781 H=2781:00 +CPU= 34.246
  01:45:37 STEP 2782 H=2782:00 +CPU= 23.688
  01:46:06 STEP 2783 H=2783:00 +CPU= 23.671
  01:46:55 STEP 2784 H=2784:00 +CPU= 43.166
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002784
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002784
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002760
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002772
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002772
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002772
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_115.zip
Uploading the intermediate file: upload_file_115.zip
  01:47:20 STEP 2785 H=2785:00 +CPU= 22.815
  01:47:46 STEP 2786 H=2786:00 +CPU= 22.515
  01:48:23 STEP 2787 H=2787:00 +CPU= 33.576
  01:48:52 STEP 2788 H=2788:00 +CPU= 23.062
  01:49:17 STEP 2789 H=2789:00 +CPU= 22.043
  01:49:55 STEP 2790 H=2790:00 +CPU= 32.424
  01:50:19 STEP 2791 H=2791:00 +CPU= 21.244
  01:50:46 STEP 2792 H=2792:00 +CPU= 23.958
  01:51:28 STEP 2793 H=2793:00 +CPU= 35.854
  01:51:56 STEP 2794 H=2794:00 +CPU= 23.414
  01:52:23 STEP 2795 H=2795:00 +CPU= 22.432
  01:53:09 STEP 2796 H=2796:00 +CPU= 38.096
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002796
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002796
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002796
  01:53:34 STEP 2797 H=2797:00 +CPU= 22.087
  01:53:58 STEP 2798 H=2798:00 +CPU= 21.796
  01:54:39 STEP 2799 H=2799:00 +CPU= 36.349
  01:55:08 STEP 2800 H=2800:00 +CPU= 24.096
  01:55:33 STEP 2801 H=2801:00 +CPU= 20.530
  01:56:11 STEP 2802 H=2802:00 +CPU= 34.694
  01:56:36 STEP 2803 H=2803:00 +CPU= 22.726
  01:57:07 STEP 2804 H=2804:00 +CPU= 24.740
  01:57:47 STEP 2805 H=2805:00 +CPU= 33.739
  01:58:11 STEP 2806 H=2806:00 +CPU= 22.549
  01:58:38 STEP 2807 H=2807:00 +CPU= 22.611
  01:59:18 STEP 2808 H=2808:00 +CPU= 37.740
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002808
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002808
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002784
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002796
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002796
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002796
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_116.zip
Uploading the intermediate file: upload_file_116.zip
  01:59:43 STEP 2809 H=2809:00 +CPU= 22.551
  02:00:08 STEP 2810 H=2810:00 +CPU= 20.110
  02:00:50 STEP 2811 H=2811:00 +CPU= 35.413
  02:01:16 STEP 2812 H=2812:00 +CPU= 23.061
  02:01:42 STEP 2813 H=2813:00 +CPU= 22.702
  02:02:23 STEP 2814 H=2814:00 +CPU= 37.025
  02:02:49 STEP 2815 H=2815:00 +CPU= 22.359
  02:03:12 STEP 2816 H=2816:00 +CPU= 21.180
  02:03:49 STEP 2817 H=2817:00 +CPU= 33.160
  02:04:16 STEP 2818 H=2818:00 +CPU= 23.291
  02:04:36 STEP 2819 H=2819:00 +CPU= 19.332
  02:05:21 STEP 2820 H=2820:00 +CPU= 37.246
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002820
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002820
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002820
  02:05:45 STEP 2821 H=2821:00 +CPU= 21.372
  02:06:10 STEP 2822 H=2822:00 +CPU= 21.832
  02:06:51 STEP 2823 H=2823:00 +CPU= 36.267
  02:07:15 STEP 2824 H=2824:00 +CPU= 23.043
  02:07:41 STEP 2825 H=2825:00 +CPU= 21.724
  02:08:20 STEP 2826 H=2826:00 +CPU= 35.467
  02:08:51 STEP 2827 H=2827:00 +CPU= 23.954
  02:09:20 STEP 2828 H=2828:00 +CPU= 23.549
  02:10:03 STEP 2829 H=2829:00 +CPU= 34.778
  02:10:29 STEP 2830 H=2830:00 +CPU= 22.594
  02:10:54 STEP 2831 H=2831:00 +CPU= 22.972
  02:11:43 STEP 2832 H=2832:00 +CPU= 42.898
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002832
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002832
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002808
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002820
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002820
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002820
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_117.zip
Uploading the intermediate file: upload_file_117.zip
  02:12:08 STEP 2833 H=2833:00 +CPU= 23.175
  02:12:35 STEP 2834 H=2834:00 +CPU= 23.018
  02:13:18 STEP 2835 H=2835:00 +CPU= 36.783
  02:13:47 STEP 2836 H=2836:00 +CPU= 23.992
  02:14:17 STEP 2837 H=2837:00 +CPU= 24.061
  02:14:57 STEP 2838 H=2838:00 +CPU= 33.889
  02:15:23 STEP 2839 H=2839:00 +CPU= 22.557
  02:15:48 STEP 2840 H=2840:00 +CPU= 22.773
  02:16:27 STEP 2841 H=2841:00 +CPU= 35.565
  02:16:54 STEP 2842 H=2842:00 +CPU= 23.520
  02:17:18 STEP 2843 H=2843:00 +CPU= 22.368
  02:18:04 STEP 2844 H=2844:00 +CPU= 38.366
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002844
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002844
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002844
  02:18:31 STEP 2845 H=2845:00 +CPU= 22.322
  02:18:54 STEP 2846 H=2846:00 +CPU= 21.075
  02:19:37 STEP 2847 H=2847:00 +CPU= 35.546
  02:20:02 STEP 2848 H=2848:00 +CPU= 23.472
  02:20:28 STEP 2849 H=2849:00 +CPU= 22.841
  02:21:09 STEP 2850 H=2850:00 +CPU= 35.635
  02:21:34 STEP 2851 H=2851:00 +CPU= 21.606
  02:21:58 STEP 2852 H=2852:00 +CPU= 22.619
  02:22:39 STEP 2853 H=2853:00 +CPU= 35.182
  02:23:05 STEP 2854 H=2854:00 +CPU= 22.311
  02:23:33 STEP 2855 H=2855:00 +CPU= 25.060
  02:24:24 STEP 2856 H=2856:00 +CPU= 42.813
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002856
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002856
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002832
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002844
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002844
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002844
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_118.zip
Uploading the intermediate file: upload_file_118.zip
  02:24:47 STEP 2857 H=2857:00 +CPU= 21.349
  02:25:15 STEP 2858 H=2858:00 +CPU= 23.044
  02:25:58 STEP 2859 H=2859:00 +CPU= 34.685
  02:26:23 STEP 2860 H=2860:00 +CPU= 23.954
  02:26:50 STEP 2861 H=2861:00 +CPU= 22.494
  02:27:29 STEP 2862 H=2862:00 +CPU= 36.318
  02:27:56 STEP 2863 H=2863:00 +CPU= 22.881
  02:28:26 STEP 2864 H=2864:00 +CPU= 24.282
  02:29:08 STEP 2865 H=2865:00 +CPU= 34.227
  02:29:33 STEP 2866 H=2866:00 +CPU= 22.513
  02:30:03 STEP 2867 H=2867:00 +CPU= 23.979
  02:30:51 STEP 2868 H=2868:00 +CPU= 37.660
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002868
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002868
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002868
  02:31:18 STEP 2869 H=2869:00 +CPU= 23.163
  02:31:47 STEP 2870 H=2870:00 +CPU= 23.089
  02:32:29 STEP 2871 H=2871:00 +CPU= 35.956
  02:32:59 STEP 2872 H=2872:00 +CPU= 23.451
  02:33:25 STEP 2873 H=2873:00 +CPU= 22.383
  02:34:04 STEP 2874 H=2874:00 +CPU= 34.304
  02:34:32 STEP 2875 H=2875:00 +CPU= 23.615
  02:34:57 STEP 2876 H=2876:00 +CPU= 22.386
  02:35:36 STEP 2877 H=2877:00 +CPU= 35.278
  02:35:57 STEP 2878 H=2878:00 +CPU= 20.162
  02:36:22 STEP 2879 H=2879:00 +CPU= 21.769
  02:37:08 STEP 2880 H=2880:00 +CPU= 39.968
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002880
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002880
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002856
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002868
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002868
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002868
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_119.zip
Uploading the intermediate file: upload_file_119.zip
  02:37:33 STEP 2881 H=2881:00 +CPU= 22.889
Uploading trickle at timestep: 10368000
  02:38:00 STEP 2882 H=2882:00 +CPU= 23.307
  02:38:42 STEP 2883 H=2883:00 +CPU= 34.970
  02:39:12 STEP 2884 H=2884:00 +CPU= 24.904
  02:39:37 STEP 2885 H=2885:00 +CPU= 22.361
  02:40:15 STEP 2886 H=2886:00 +CPU= 34.836
  02:40:39 STEP 2887 H=2887:00 +CPU= 22.180
  02:41:07 STEP 2888 H=2888:00 +CPU= 23.510
  02:41:45 STEP 2889 H=2889:00 +CPU= 34.363
  02:42:11 STEP 2890 H=2890:00 +CPU= 21.788
  02:42:37 STEP 2891 H=2891:00 +CPU= 22.270
  02:43:19 STEP 2892 H=2892:00 +CPU= 37.502
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002892
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002892
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002892
  02:43:45 STEP 2893 H=2893:00 +CPU= 22.552
  02:44:12 STEP 2894 H=2894:00 +CPU= 22.450
  02:44:48 STEP 2895 H=2895:00 +CPU= 33.994
  02:45:13 STEP 2896 H=2896:00 +CPU= 22.172
  02:45:43 STEP 2897 H=2897:00 +CPU= 22.754
  02:46:21 STEP 2898 H=2898:00 +CPU= 34.745
  02:46:49 STEP 2899 H=2899:00 +CPU= 22.455
  02:47:17 STEP 2900 H=2900:00 +CPU= 24.869
  02:47:56 STEP 2901 H=2901:00 +CPU= 33.682
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0558
(argv4) batchid: 989
(argv5) wuid: 12206202
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/jeff/slots/14
Project directory is: /home/jeff/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202
..mkdir for temp folder for results failed
Copying: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/jeff/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/jeff/slots/14/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_bc77a2833678c76fe77e385a8ea6b5b5 to: /home/jeff/slots/14/oifs_43r3_ps_0558_2020050100_123_989_12206202.zip
Unzipping the namelist zip file: /home/jeff/slots/14/oifs_43r3_ps_0558_2020050100_123_989_12206202.zip
ic_ancil_file: ic_ancil_12206202
ifsdata_file: ifsdata_12206202
climate_data_file: clim_data_12206202
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9fca43bbc4286d19ab5714e43b24d6a7 to: /home/jeff/slots/14/ic_ancil_12206202.zip
Unzipping the IC ancils zip file: /home/jeff/slots/14/ic_ancil_12206202.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8bc04831b2c69ce01bb4df43c3100d30 to: /home/jeff/slots/14/ifsdata/ifsdata_12206202.zip
Unzipping the ifsdata_zip file: /home/jeff/slots/14/ifsdata/ifsdata_12206202.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/jeff/slots/14/159l_2/clim_data_12206202.zip
Unzipping the climate data zip file: /home/jeff/slots/14/159l_2/clim_data_12206202.zip
Checking for progress XML file: /home/jeff/slots/14/progress_file_12206202.xml
Opened progress file ok : /home/jeff/slots/14/progress_file_12206202.xml
-- Model is restarting --
Adjusting last_iter, 2902, to previous model restart step.
Creating progress file: /home/jeff/slots/14/progress_file_12206202.xml
last_cpu_time: 81978
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061
The child process has been launched with process id: 10855
Executing the command: /home/jeff/slots/14/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at 0x1)
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Cthulhu:1:1:10855:10855] [20230114:004350:1673678630:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:44:44 STEP 2880 H=2880:00 +CPU= 36.339
  00:45:08 STEP 2881 H=2881:00 +CPU= 21.874
  00:45:34 STEP 2882 H=2882:00 +CPU= 23.446
  00:46:15 STEP 2883 H=2883:00 +CPU= 34.327
  00:46:42 STEP 2884 H=2884:00 +CPU= 23.867
  00:47:12 STEP 2885 H=2885:00 +CPU= 24.507
  00:47:53 STEP 2886 H=2886:00 +CPU= 36.599
  00:48:22 STEP 2887 H=2887:00 +CPU= 24.104
  00:48:50 STEP 2888 H=2888:00 +CPU= 23.363
  00:49:36 STEP 2889 H=2889:00 +CPU= 36.458
  00:50:04 STEP 2890 H=2890:00 +CPU= 23.901
  00:50:30 STEP 2891 H=2891:00 +CPU= 24.377
  00:51:13 STEP 2892 H=2892:00 +CPU= 37.480
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002892
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002892
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002892
  00:51:41 STEP 2893 H=2893:00 +CPU= 23.886
  00:52:10 STEP 2894 H=2894:00 +CPU= 23.979
  00:52:49 STEP 2895 H=2895:00 +CPU= 35.262
  00:53:15 STEP 2896 H=2896:00 +CPU= 25.043
  00:53:47 STEP 2897 H=2897:00 +CPU= 24.571
  00:54:29 STEP 2898 H=2898:00 +CPU= 37.249
  00:54:54 STEP 2899 H=2899:00 +CPU= 22.882
  00:55:21 STEP 2900 H=2900:00 +CPU= 23.970
  00:56:00 STEP 2901 H=2901:00 +CPU= 35.195
  00:56:28 STEP 2902 H=2902:00 +CPU= 24.181
  00:56:56 STEP 2903 H=2903:00 +CPU= 24.761
  00:57:47 STEP 2904 H=2904:00 +CPU= 42.649
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002904
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002904
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002880
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002892
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002892
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002892
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_120.zip
Uploading the intermediate file: upload_file_120.zip
  00:58:15 STEP 2905 H=2905:00 +CPU= 23.840
  00:58:40 STEP 2906 H=2906:00 +CPU= 23.673
  00:59:19 STEP 2907 H=2907:00 +CPU= 35.300
  00:59:44 STEP 2908 H=2908:00 +CPU= 23.550
  01:00:11 STEP 2909 H=2909:00 +CPU= 24.328
  01:00:55 STEP 2910 H=2910:00 +CPU= 38.136
  01:01:26 STEP 2911 H=2911:00 +CPU= 24.704
  01:01:51 STEP 2912 H=2912:00 +CPU= 22.019
  01:02:31 STEP 2913 H=2913:00 +CPU= 36.384
  01:02:58 STEP 2914 H=2914:00 +CPU= 24.027
  01:03:24 STEP 2915 H=2915:00 +CPU= 22.958
  01:04:09 STEP 2916 H=2916:00 +CPU= 40.258
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002916
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002916
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002916
  01:04:33 STEP 2917 H=2917:00 +CPU= 22.066
  01:05:03 STEP 2918 H=2918:00 +CPU= 24.384
  01:05:44 STEP 2919 H=2919:00 +CPU= 36.238
  01:06:11 STEP 2920 H=2920:00 +CPU= 23.036
  01:06:38 STEP 2921 H=2921:00 +CPU= 24.213
  01:07:22 STEP 2922 H=2922:00 +CPU= 37.103
  01:07:52 STEP 2923 H=2923:00 +CPU= 24.637
  01:08:20 STEP 2924 H=2924:00 +CPU= 24.454
  01:09:01 STEP 2925 H=2925:00 +CPU= 36.390
  01:09:29 STEP 2926 H=2926:00 +CPU= 24.822
  01:09:58 STEP 2927 H=2927:00 +CPU= 24.699
  01:10:42 STEP 2928 H=2928:00 +CPU= 40.622
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002928
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002928
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002928
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002904
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002916
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002916
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002916
Zipping up the intermediate file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_121.zip
Uploading the intermediate file: upload_file_121.zip
  01:11:05 STEP 2929 H=2929:00 +CPU= 22.833
  01:11:30 STEP 2930 H=2930:00 +CPU= 24.257
  01:12:07 STEP 2931 H=2931:00 +CPU= 36.303
  01:12:31 STEP 2932 H=2932:00 +CPU= 23.148
  01:12:54 STEP 2933 H=2933:00 +CPU= 22.842
  01:13:31 STEP 2934 H=2934:00 +CPU= 36.785
  01:13:53 STEP 2935 H=2935:00 +CPU= 21.811
  01:14:19 STEP 2936 H=2936:00 +CPU= 24.093
  01:14:58 STEP 2937 H=2937:00 +CPU= 35.193
  01:15:25 STEP 2938 H=2938:00 +CPU= 24.267
  01:15:48 STEP 2939 H=2939:00 +CPU= 21.585
  01:16:32 STEP 2940 H=2940:00 +CPU= 38.905
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002940
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002940
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002940
  01:16:58 STEP 2941 H=2941:00 +CPU= 23.814
  01:17:26 STEP 2942 H=2942:00 +CPU= 24.594
  01:18:05 STEP 2943 H=2943:00 +CPU= 35.085
  01:18:32 STEP 2944 H=2944:00 +CPU= 24.558
  01:18:58 STEP 2945 H=2945:00 +CPU= 21.495
  01:19:42 STEP 2946 H=2946:00 +CPU= 35.438
  01:20:11 STEP 2947 H=2947:00 +CPU= 23.799
  01:20:37 STEP 2948 H=2948:00 +CPU= 23.904
  01:21:17 STEP 2949 H=2949:00 +CPU= 35.036
  01:21:45 STEP 2950 H=2950:00 +CPU= 24.069
  01:22:11 STEP 2951 H=2951:00 +CPU= 24.449
  01:22:55 STEP 2952 H=2952:00 +CPU= 40.320
The child process terminated with status: 0
Moving to projects directory: /home/jeff/slots/14/ICMGGhq0f+002952
Moving to projects directory: /home/jeff/slots/14/ICMSHhq0f+002952
Moving to projects directory: /home/jeff/slots/14/ICMUAhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002928
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002928
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002940
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002940
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMSHhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMGGhq0f+002940
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002952
Adding to the zip: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_12206202/ICMUAhq0f+002928
Zipping up the final file: /home/jeff/projects/climateprediction.net/oifs_43r3_ps_0558_2020050100_123_989_12206202_0_r596351061_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
01:26:19 (10821): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:18:58 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 10,623,600 84,045 0.0079
09 May 2023 16:17:50 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 10,368,000 81,398 0.0079
09 May 2023 16:17:43 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 9,504,000 74,932 0.0079
09 May 2023 16:17:37 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 8,640,000 68,490 0.0079
09 May 2023 16:17:29 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 7,776,000 61,894 0.0080
09 May 2023 16:17:20 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 6,912,000 54,802 0.0079
09 May 2023 16:17:10 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 6,048,000 47,543 0.0079
09 May 2023 16:17:03 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 5,184,000 40,388 0.0078
09 May 2023 16:16:52 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 4,320,000 33,470 0.0077
09 May 2023 16:16:42 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 3,456,000 26,457 0.0077
09 May 2023 16:16:33 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 2,592,000 19,724 0.0076
09 May 2023 16:16:26 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 1,728,000 13,283 0.0077
09 May 2023 16:16:18 1538024 22292792 oifs_43r3_ps_0558_2020050100_123_989_12206202_0 864,000 6,806 0.0079


©2024 cpdn.org