climateprediction.net home page
Task 22270037

Task 22270037

Name oifs_43r3_ps_0125_1999050100_123_968_12184769_0
Workunit 12184769
Created 22 Dec 2022, 18:00:13 UTC
Sent 17 Jan 2023, 23:23:22 UTC
Report deadline 18 Mar 2023, 23:23:22 UTC
Received 18 Jan 2023, 22:04:45 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1538236
Run time 21 hours 49 min 39 sec
CPU time 21 hours 44 min 13 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 5.98 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.70 MB
Peak swap size 4,978.05 MB
Peak disk usage 2,595.28 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
ar/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_104.zip
Uploading the intermediate file: upload_file_104.zip
  17:47:11 STEP 2521 H=2521:00 +CPU= 27.873
  17:47:38 STEP 2522 H=2522:00 +CPU= 26.364
  17:48:16 STEP 2523 H=2523:00 +CPU= 38.402
  17:48:43 STEP 2524 H=2524:00 +CPU= 26.908
  17:49:09 STEP 2525 H=2525:00 +CPU= 25.562
  17:49:49 STEP 2526 H=2526:00 +CPU= 40.410
  17:50:14 STEP 2527 H=2527:00 +CPU= 24.586
  17:50:40 STEP 2528 H=2528:00 +CPU= 26.340
  17:51:15 STEP 2529 H=2529:00 +CPU= 35.151
  17:51:40 STEP 2530 H=2530:00 +CPU= 24.760
  17:52:02 STEP 2531 H=2531:00 +CPU= 21.767
  17:52:41 STEP 2532 H=2532:00 +CPU= 39.455
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002532
  17:53:04 STEP 2533 H=2533:00 +CPU= 22.445
  17:53:29 STEP 2534 H=2534:00 +CPU= 25.380
  17:54:09 STEP 2535 H=2535:00 +CPU= 39.857
  17:54:35 STEP 2536 H=2536:00 +CPU= 25.705
  17:54:59 STEP 2537 H=2537:00 +CPU= 24.233
  17:55:35 STEP 2538 H=2538:00 +CPU= 35.987
  17:56:00 STEP 2539 H=2539:00 +CPU= 24.656
  17:56:26 STEP 2540 H=2540:00 +CPU= 25.906
  17:57:05 STEP 2541 H=2541:00 +CPU= 39.842
  17:57:31 STEP 2542 H=2542:00 +CPU= 25.438
  17:57:51 STEP 2543 H=2543:00 +CPU= 19.777
  17:58:37 STEP 2544 H=2544:00 +CPU= 46.308
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_105.zip
Uploading the intermediate file: upload_file_105.zip
  17:59:00 STEP 2545 H=2545:00 +CPU= 23.414
  17:59:28 STEP 2546 H=2546:00 +CPU= 27.171
  18:00:07 STEP 2547 H=2547:00 +CPU= 39.429
  18:00:32 STEP 2548 H=2548:00 +CPU= 25.170
  18:00:55 STEP 2549 H=2549:00 +CPU= 22.802
  18:01:32 STEP 2550 H=2550:00 +CPU= 37.463
  18:01:56 STEP 2551 H=2551:00 +CPU= 23.952
  18:02:23 STEP 2552 H=2552:00 +CPU= 26.288
  18:03:00 STEP 2553 H=2553:00 +CPU= 36.859
  18:03:22 STEP 2554 H=2554:00 +CPU= 22.674
  18:03:46 STEP 2555 H=2555:00 +CPU= 23.651
  18:04:31 STEP 2556 H=2556:00 +CPU= 44.535
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002556
  18:04:56 STEP 2557 H=2557:00 +CPU= 25.494
  18:05:21 STEP 2558 H=2558:00 +CPU= 24.773
  18:06:00 STEP 2559 H=2559:00 +CPU= 38.979
  18:06:25 STEP 2560 H=2560:00 +CPU= 25.103
  18:06:49 STEP 2561 H=2561:00 +CPU= 23.837
  18:07:31 STEP 2562 H=2562:00 +CPU= 41.811
  18:07:56 STEP 2563 H=2563:00 +CPU= 24.822
  18:08:22 STEP 2564 H=2564:00 +CPU= 26.698
  18:09:00 STEP 2565 H=2565:00 +CPU= 37.806
  18:09:25 STEP 2566 H=2566:00 +CPU= 25.240
  18:09:53 STEP 2567 H=2567:00 +CPU= 27.812
  18:10:40 STEP 2568 H=2568:00 +CPU= 46.953
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_106.zip
Uploading the intermediate file: upload_file_106.zip
  18:11:06 STEP 2569 H=2569:00 +CPU= 26.054
  18:11:34 STEP 2570 H=2570:00 +CPU= 27.381
  18:12:15 STEP 2571 H=2571:00 +CPU= 41.068
  18:12:43 STEP 2572 H=2572:00 +CPU= 27.909
  18:13:10 STEP 2573 H=2573:00 +CPU= 27.342
  18:13:52 STEP 2574 H=2574:00 +CPU= 42.452
  18:14:18 STEP 2575 H=2575:00 +CPU= 25.759
  18:14:45 STEP 2576 H=2576:00 +CPU= 26.448
  18:15:23 STEP 2577 H=2577:00 +CPU= 37.917
  18:15:48 STEP 2578 H=2578:00 +CPU= 25.840
  18:16:15 STEP 2579 H=2579:00 +CPU= 26.713
  18:16:59 STEP 2580 H=2580:00 +CPU= 44.278
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002580
  18:17:25 STEP 2581 H=2581:00 +CPU= 25.956
  18:17:50 STEP 2582 H=2582:00 +CPU= 24.922
  18:18:28 STEP 2583 H=2583:00 +CPU= 37.284
  18:18:55 STEP 2584 H=2584:00 +CPU= 27.196
  18:19:20 STEP 2585 H=2585:00 +CPU= 25.145
  18:20:00 STEP 2586 H=2586:00 +CPU= 39.632
  18:20:24 STEP 2587 H=2587:00 +CPU= 24.181
  18:20:48 STEP 2588 H=2588:00 +CPU= 23.961
  18:21:24 STEP 2589 H=2589:00 +CPU= 36.398
  18:21:47 STEP 2590 H=2590:00 +CPU= 23.296
  18:22:11 STEP 2591 H=2591:00 +CPU= 23.215
  18:22:49 STEP 2592 H=2592:00 +CPU= 38.140
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_107.zip
Uploading the intermediate file: upload_file_107.zip
  18:23:11 STEP 2593 H=2593:00 +CPU= 21.816
  18:23:36 STEP 2594 H=2594:00 +CPU= 25.157
  18:24:13 STEP 2595 H=2595:00 +CPU= 37.064
  18:24:36 STEP 2596 H=2596:00 +CPU= 23.052
  18:25:01 STEP 2597 H=2597:00 +CPU= 25.047
  18:25:41 STEP 2598 H=2598:00 +CPU= 40.015
  18:26:04 STEP 2599 H=2599:00 +CPU= 23.470
  18:26:29 STEP 2600 H=2600:00 +CPU= 24.215
  18:27:01 STEP 2601 H=2601:00 +CPU= 32.563
  18:27:24 STEP 2602 H=2602:00 +CPU= 23.012
  18:27:49 STEP 2603 H=2603:00 +CPU= 24.451
  18:28:32 STEP 2604 H=2604:00 +CPU= 42.644
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002604
  18:28:57 STEP 2605 H=2605:00 +CPU= 25.419
  18:29:22 STEP 2606 H=2606:00 +CPU= 25.076
  18:30:01 STEP 2607 H=2607:00 +CPU= 38.919
  18:30:25 STEP 2608 H=2608:00 +CPU= 23.827
  18:30:47 STEP 2609 H=2609:00 +CPU= 21.711
  18:31:23 STEP 2610 H=2610:00 +CPU= 36.107
  18:31:42 STEP 2611 H=2611:00 +CPU= 19.589
  18:32:03 STEP 2612 H=2612:00 +CPU= 20.978
  18:32:37 STEP 2613 H=2613:00 +CPU= 33.931
  18:32:59 STEP 2614 H=2614:00 +CPU= 21.439
  18:33:25 STEP 2615 H=2615:00 +CPU= 25.992
  18:34:03 STEP 2616 H=2616:00 +CPU= 38.397
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_108.zip
Uploading the intermediate file: upload_file_108.zip
  18:34:29 STEP 2617 H=2617:00 +CPU= 25.979
  18:34:54 STEP 2618 H=2618:00 +CPU= 25.185
  18:35:34 STEP 2619 H=2619:00 +CPU= 40.026
  18:36:00 STEP 2620 H=2620:00 +CPU= 25.933
  18:36:24 STEP 2621 H=2621:00 +CPU= 23.423
  18:37:01 STEP 2622 H=2622:00 +CPU= 37.398
  18:37:23 STEP 2623 H=2623:00 +CPU= 22.315
  18:37:50 STEP 2624 H=2624:00 +CPU= 26.954
  18:38:32 STEP 2625 H=2625:00 +CPU= 41.053
  18:38:55 STEP 2626 H=2626:00 +CPU= 23.797
  18:39:21 STEP 2627 H=2627:00 +CPU= 25.337
  18:40:05 STEP 2628 H=2628:00 +CPU= 43.980
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002628
  18:40:30 STEP 2629 H=2629:00 +CPU= 25.067
  18:40:49 STEP 2630 H=2630:00 +CPU= 19.479
  18:41:27 STEP 2631 H=2631:00 +CPU= 37.773
  18:41:52 STEP 2632 H=2632:00 +CPU= 24.796
  18:42:18 STEP 2633 H=2633:00 +CPU= 25.676
  18:42:57 STEP 2634 H=2634:00 +CPU= 39.352
  18:43:22 STEP 2635 H=2635:00 +CPU= 24.686
  18:43:48 STEP 2636 H=2636:00 +CPU= 25.926
  18:44:26 STEP 2637 H=2637:00 +CPU= 37.903
  18:44:52 STEP 2638 H=2638:00 +CPU= 26.247
  18:45:18 STEP 2639 H=2639:00 +CPU= 26.319
  18:46:02 STEP 2640 H=2640:00 +CPU= 43.348
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_109.zip
Uploading the intermediate file: upload_file_109.zip
Uploading trickle at timestep: 9504000
  18:46:27 STEP 2641 H=2641:00 +CPU= 25.047
  18:46:52 STEP 2642 H=2642:00 +CPU= 25.130
  18:47:29 STEP 2643 H=2643:00 +CPU= 37.524
  18:47:53 STEP 2644 H=2644:00 +CPU= 23.845
  18:48:15 STEP 2645 H=2645:00 +CPU= 22.140
  18:48:52 STEP 2646 H=2646:00 +CPU= 36.632
  18:49:17 STEP 2647 H=2647:00 +CPU= 25.041
  18:49:43 STEP 2648 H=2648:00 +CPU= 26.415
  18:50:24 STEP 2649 H=2649:00 +CPU= 40.392
  18:50:47 STEP 2650 H=2650:00 +CPU= 23.189
  18:51:09 STEP 2651 H=2651:00 +CPU= 21.906
  18:51:49 STEP 2652 H=2652:00 +CPU= 40.131
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002652
  18:52:11 STEP 2653 H=2653:00 +CPU= 21.832
  18:52:36 STEP 2654 H=2654:00 +CPU= 25.374
  18:53:16 STEP 2655 H=2655:00 +CPU= 39.488
  18:53:42 STEP 2656 H=2656:00 +CPU= 26.073
  18:54:06 STEP 2657 H=2657:00 +CPU= 24.374
  18:54:47 STEP 2658 H=2658:00 +CPU= 41.033
  18:55:13 STEP 2659 H=2659:00 +CPU= 25.203
  18:55:40 STEP 2660 H=2660:00 +CPU= 27.295
  18:56:22 STEP 2661 H=2661:00 +CPU= 42.041
  18:56:45 STEP 2662 H=2662:00 +CPU= 23.174
  18:57:10 STEP 2663 H=2663:00 +CPU= 25.337
  18:57:56 STEP 2664 H=2664:00 +CPU= 45.704
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_110.zip
Uploading the intermediate file: upload_file_110.zip
  18:58:21 STEP 2665 H=2665:00 +CPU= 24.370
  18:58:48 STEP 2666 H=2666:00 +CPU= 27.011
  18:59:20 STEP 2667 H=2667:00 +CPU= 32.648
  18:59:44 STEP 2668 H=2668:00 +CPU= 24.044
  19:00:07 STEP 2669 H=2669:00 +CPU= 23.181
  19:00:45 STEP 2670 H=2670:00 +CPU= 37.419
  19:01:09 STEP 2671 H=2671:00 +CPU= 23.606
  19:01:30 STEP 2672 H=2672:00 +CPU= 21.042
  19:02:05 STEP 2673 H=2673:00 +CPU= 35.750
  19:02:28 STEP 2674 H=2674:00 +CPU= 22.585
  19:02:53 STEP 2675 H=2675:00 +CPU= 25.070
  19:03:36 STEP 2676 H=2676:00 +CPU= 42.891
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002676
  19:04:03 STEP 2677 H=2677:00 +CPU= 26.411
  19:04:28 STEP 2678 H=2678:00 +CPU= 25.563
  19:05:08 STEP 2679 H=2679:00 +CPU= 39.951
  19:05:34 STEP 2680 H=2680:00 +CPU= 26.204
  19:05:59 STEP 2681 H=2681:00 +CPU= 24.558
  19:06:35 STEP 2682 H=2682:00 +CPU= 35.599
  19:06:56 STEP 2683 H=2683:00 +CPU= 21.964
  19:07:18 STEP 2684 H=2684:00 +CPU= 21.246
  19:07:53 STEP 2685 H=2685:00 +CPU= 34.900
  19:08:17 STEP 2686 H=2686:00 +CPU= 24.742
  19:08:42 STEP 2687 H=2687:00 +CPU= 24.292
  19:09:26 STEP 2688 H=2688:00 +CPU= 43.886
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_111.zip
Uploading the intermediate file: upload_file_111.zip
  19:09:50 STEP 2689 H=2689:00 +CPU= 23.961
  19:10:14 STEP 2690 H=2690:00 +CPU= 24.621
  19:10:50 STEP 2691 H=2691:00 +CPU= 36.124
  19:11:15 STEP 2692 H=2692:00 +CPU= 24.490
  19:11:38 STEP 2693 H=2693:00 +CPU= 22.904
  19:12:15 STEP 2694 H=2694:00 +CPU= 37.172
  19:12:39 STEP 2695 H=2695:00 +CPU= 24.467
  19:13:04 STEP 2696 H=2696:00 +CPU= 25.143
  19:13:40 STEP 2697 H=2697:00 +CPU= 35.865
  19:14:04 STEP 2698 H=2698:00 +CPU= 23.177
  19:14:27 STEP 2699 H=2699:00 +CPU= 22.989
  19:15:08 STEP 2700 H=2700:00 +CPU= 41.710
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002700
  19:15:32 STEP 2701 H=2701:00 +CPU= 23.980
  19:15:56 STEP 2702 H=2702:00 +CPU= 24.155
  19:16:31 STEP 2703 H=2703:00 +CPU= 34.568
  19:16:55 STEP 2704 H=2704:00 +CPU= 24.023
  19:17:19 STEP 2705 H=2705:00 +CPU= 24.184
  19:18:06 STEP 2706 H=2706:00 +CPU= 46.418
  19:18:32 STEP 2707 H=2707:00 +CPU= 26.760
  19:18:59 STEP 2708 H=2708:00 +CPU= 26.023
  19:19:37 STEP 2709 H=2709:00 +CPU= 38.322
  19:20:01 STEP 2710 H=2710:00 +CPU= 24.217
  19:20:24 STEP 2711 H=2711:00 +CPU= 23.423
  19:21:03 STEP 2712 H=2712:00 +CPU= 38.090
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_112.zip
Uploading the intermediate file: upload_file_112.zip
  19:21:26 STEP 2713 H=2713:00 +CPU= 23.063
  19:21:48 STEP 2714 H=2714:00 +CPU= 22.283
  19:22:22 STEP 2715 H=2715:00 +CPU= 34.329
  19:22:43 STEP 2716 H=2716:00 +CPU= 20.719
  19:23:04 STEP 2717 H=2717:00 +CPU= 20.978
  19:23:36 STEP 2718 H=2718:00 +CPU= 31.993
  19:23:56 STEP 2719 H=2719:00 +CPU= 19.444
  19:24:17 STEP 2720 H=2720:00 +CPU= 21.789
  19:24:52 STEP 2721 H=2721:00 +CPU= 34.355
  19:25:14 STEP 2722 H=2722:00 +CPU= 21.733
  19:25:37 STEP 2723 H=2723:00 +CPU= 23.186
  19:26:16 STEP 2724 H=2724:00 +CPU= 38.970
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002724
  19:26:40 STEP 2725 H=2725:00 +CPU= 23.769
  19:27:02 STEP 2726 H=2726:00 +CPU= 22.570
  19:27:40 STEP 2727 H=2727:00 +CPU= 38.107
  19:28:04 STEP 2728 H=2728:00 +CPU= 23.759
  19:28:27 STEP 2729 H=2729:00 +CPU= 22.988
  19:28:59 STEP 2730 H=2730:00 +CPU= 31.520
  19:29:19 STEP 2731 H=2731:00 +CPU= 19.955
  19:29:41 STEP 2732 H=2732:00 +CPU= 22.324
  19:30:14 STEP 2733 H=2733:00 +CPU= 32.955
  19:30:42 STEP 2734 H=2734:00 +CPU= 27.627
  19:31:09 STEP 2735 H=2735:00 +CPU= 27.438
  19:31:51 STEP 2736 H=2736:00 +CPU= 41.988
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_113.zip
Uploading the intermediate file: upload_file_113.zip
  19:32:11 STEP 2737 H=2737:00 +CPU= 20.383
  19:32:35 STEP 2738 H=2738:00 +CPU= 23.561
  19:33:12 STEP 2739 H=2739:00 +CPU= 37.462
  19:33:35 STEP 2740 H=2740:00 +CPU= 22.524
  19:34:00 STEP 2741 H=2741:00 +CPU= 24.592
  19:34:39 STEP 2742 H=2742:00 +CPU= 39.157
  19:35:01 STEP 2743 H=2743:00 +CPU= 21.967
  19:35:22 STEP 2744 H=2744:00 +CPU= 21.521
  19:35:57 STEP 2745 H=2745:00 +CPU= 35.237
  19:36:20 STEP 2746 H=2746:00 +CPU= 22.733
  19:36:45 STEP 2747 H=2747:00 +CPU= 24.511
  19:37:27 STEP 2748 H=2748:00 +CPU= 42.491
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002748
  19:37:51 STEP 2749 H=2749:00 +CPU= 24.141
  19:38:10 STEP 2750 H=2750:00 +CPU= 18.532
  19:38:47 STEP 2751 H=2751:00 +CPU= 36.789
  19:39:10 STEP 2752 H=2752:00 +CPU= 23.617
  19:39:34 STEP 2753 H=2753:00 +CPU= 23.153
  19:40:09 STEP 2754 H=2754:00 +CPU= 35.326
  19:40:33 STEP 2755 H=2755:00 +CPU= 24.323
  19:40:59 STEP 2756 H=2756:00 +CPU= 25.647
  19:41:34 STEP 2757 H=2757:00 +CPU= 35.453
  19:41:58 STEP 2758 H=2758:00 +CPU= 23.279
  19:42:25 STEP 2759 H=2759:00 +CPU= 27.806
  19:43:18 STEP 2760 H=2760:00 +CPU= 52.404
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_114.zip
Uploading the intermediate file: upload_file_114.zip
  19:43:42 STEP 2761 H=2761:00 +CPU= 24.059
  19:44:07 STEP 2762 H=2762:00 +CPU= 24.991
  19:44:46 STEP 2763 H=2763:00 +CPU= 38.691
  19:45:09 STEP 2764 H=2764:00 +CPU= 22.972
  19:45:30 STEP 2765 H=2765:00 +CPU= 21.103
  19:46:04 STEP 2766 H=2766:00 +CPU= 33.924
  19:46:28 STEP 2767 H=2767:00 +CPU= 23.869
  19:46:50 STEP 2768 H=2768:00 +CPU= 22.665
  19:47:26 STEP 2769 H=2769:00 +CPU= 35.322
  19:47:48 STEP 2770 H=2770:00 +CPU= 22.264
  19:48:12 STEP 2771 H=2771:00 +CPU= 24.385
  19:48:51 STEP 2772 H=2772:00 +CPU= 38.587
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002772
  19:49:15 STEP 2773 H=2773:00 +CPU= 23.725
  19:49:39 STEP 2774 H=2774:00 +CPU= 24.227
  19:50:14 STEP 2775 H=2775:00 +CPU= 35.566
  19:50:38 STEP 2776 H=2776:00 +CPU= 23.127
  19:51:01 STEP 2777 H=2777:00 +CPU= 23.470
  19:51:39 STEP 2778 H=2778:00 +CPU= 37.897
  19:52:01 STEP 2779 H=2779:00 +CPU= 22.471
  19:52:26 STEP 2780 H=2780:00 +CPU= 24.336
  19:53:02 STEP 2781 H=2781:00 +CPU= 36.708
  19:53:25 STEP 2782 H=2782:00 +CPU= 22.432
  19:53:47 STEP 2783 H=2783:00 +CPU= 21.914
  19:54:25 STEP 2784 H=2784:00 +CPU= 37.856
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_115.zip
Uploading the intermediate file: upload_file_115.zip
  19:54:49 STEP 2785 H=2785:00 +CPU= 24.083
  19:55:15 STEP 2786 H=2786:00 +CPU= 26.350
  19:55:51 STEP 2787 H=2787:00 +CPU= 36.052
  19:56:14 STEP 2788 H=2788:00 +CPU= 22.572
  19:56:37 STEP 2789 H=2789:00 +CPU= 23.240
  19:57:17 STEP 2790 H=2790:00 +CPU= 39.443
  19:57:39 STEP 2791 H=2791:00 +CPU= 22.754
  19:58:02 STEP 2792 H=2792:00 +CPU= 23.093
  19:58:38 STEP 2793 H=2793:00 +CPU= 35.338
  19:59:02 STEP 2794 H=2794:00 +CPU= 24.491
  19:59:26 STEP 2795 H=2795:00 +CPU= 23.377
  20:00:07 STEP 2796 H=2796:00 +CPU= 40.810
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002796
  20:00:30 STEP 2797 H=2797:00 +CPU= 23.776
  20:00:55 STEP 2798 H=2798:00 +CPU= 24.282
  20:01:29 STEP 2799 H=2799:00 +CPU= 34.306
  20:01:53 STEP 2800 H=2800:00 +CPU= 24.266
  20:02:15 STEP 2801 H=2801:00 +CPU= 21.748
  20:02:49 STEP 2802 H=2802:00 +CPU= 33.972
  20:03:12 STEP 2803 H=2803:00 +CPU= 22.833
  20:03:35 STEP 2804 H=2804:00 +CPU= 23.445
  20:04:12 STEP 2805 H=2805:00 +CPU= 36.437
  20:04:36 STEP 2806 H=2806:00 +CPU= 24.252
  20:05:01 STEP 2807 H=2807:00 +CPU= 25.008
  20:05:41 STEP 2808 H=2808:00 +CPU= 39.771
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_116.zip
Uploading the intermediate file: upload_file_116.zip
  20:06:01 STEP 2809 H=2809:00 +CPU= 20.209
  20:06:25 STEP 2810 H=2810:00 +CPU= 23.845
  20:07:01 STEP 2811 H=2811:00 +CPU= 36.367
  20:07:25 STEP 2812 H=2812:00 +CPU= 23.839
  20:07:46 STEP 2813 H=2813:00 +CPU= 21.414
  20:08:24 STEP 2814 H=2814:00 +CPU= 37.212
  20:08:48 STEP 2815 H=2815:00 +CPU= 24.618
  20:09:14 STEP 2816 H=2816:00 +CPU= 25.590
  20:09:49 STEP 2817 H=2817:00 +CPU= 35.391
  20:10:12 STEP 2818 H=2818:00 +CPU= 22.390
  20:10:34 STEP 2819 H=2819:00 +CPU= 22.497
  20:11:14 STEP 2820 H=2820:00 +CPU= 39.623
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002820
  20:11:36 STEP 2821 H=2821:00 +CPU= 21.953
  20:12:00 STEP 2822 H=2822:00 +CPU= 23.874
  20:12:35 STEP 2823 H=2823:00 +CPU= 35.677
  20:13:01 STEP 2824 H=2824:00 +CPU= 25.097
  20:13:23 STEP 2825 H=2825:00 +CPU= 22.560
  20:14:00 STEP 2826 H=2826:00 +CPU= 36.860
  20:14:23 STEP 2827 H=2827:00 +CPU= 22.741
  20:14:47 STEP 2828 H=2828:00 +CPU= 24.382
  20:15:22 STEP 2829 H=2829:00 +CPU= 35.094
  20:15:47 STEP 2830 H=2830:00 +CPU= 24.641
  20:16:10 STEP 2831 H=2831:00 +CPU= 23.110
  20:16:54 STEP 2832 H=2832:00 +CPU= 44.206
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_117.zip
Uploading the intermediate file: upload_file_117.zip
  20:17:18 STEP 2833 H=2833:00 +CPU= 23.536
  20:17:42 STEP 2834 H=2834:00 +CPU= 24.415
  20:18:18 STEP 2835 H=2835:00 +CPU= 35.566
  20:18:39 STEP 2836 H=2836:00 +CPU= 20.891
  20:19:02 STEP 2837 H=2837:00 +CPU= 22.900
  20:19:40 STEP 2838 H=2838:00 +CPU= 37.960
  20:20:03 STEP 2839 H=2839:00 +CPU= 23.439
  20:20:27 STEP 2840 H=2840:00 +CPU= 23.961
  20:21:03 STEP 2841 H=2841:00 +CPU= 35.422
  20:21:27 STEP 2842 H=2842:00 +CPU= 24.113
  20:21:50 STEP 2843 H=2843:00 +CPU= 23.458
  20:22:27 STEP 2844 H=2844:00 +CPU= 36.452
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002844
  20:22:51 STEP 2845 H=2845:00 +CPU= 24.231
  20:23:14 STEP 2846 H=2846:00 +CPU= 23.051
  20:23:50 STEP 2847 H=2847:00 +CPU= 35.667
  20:24:15 STEP 2848 H=2848:00 +CPU= 25.464
  20:24:39 STEP 2849 H=2849:00 +CPU= 24.366
  20:25:15 STEP 2850 H=2850:00 +CPU= 35.255
  20:25:40 STEP 2851 H=2851:00 +CPU= 24.862
  20:26:04 STEP 2852 H=2852:00 +CPU= 24.902
  20:26:37 STEP 2853 H=2853:00 +CPU= 32.356
  20:27:01 STEP 2854 H=2854:00 +CPU= 24.078
  20:27:26 STEP 2855 H=2855:00 +CPU= 24.981
  20:28:09 STEP 2856 H=2856:00 +CPU= 43.452
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_118.zip
Uploading the intermediate file: upload_file_118.zip
  20:28:33 STEP 2857 H=2857:00 +CPU= 23.432
  20:28:57 STEP 2858 H=2858:00 +CPU= 23.978
  20:29:32 STEP 2859 H=2859:00 +CPU= 34.750
  20:29:54 STEP 2860 H=2860:00 +CPU= 22.666
  20:30:17 STEP 2861 H=2861:00 +CPU= 22.595
  20:30:53 STEP 2862 H=2862:00 +CPU= 36.463
  20:31:16 STEP 2863 H=2863:00 +CPU= 22.549
  20:31:40 STEP 2864 H=2864:00 +CPU= 24.590
  20:32:16 STEP 2865 H=2865:00 +CPU= 35.654
  20:32:40 STEP 2866 H=2866:00 +CPU= 23.993
  20:33:04 STEP 2867 H=2867:00 +CPU= 23.346
  20:33:43 STEP 2868 H=2868:00 +CPU= 39.536
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002868
  20:34:05 STEP 2869 H=2869:00 +CPU= 21.618
  20:34:27 STEP 2870 H=2870:00 +CPU= 22.215
  20:35:01 STEP 2871 H=2871:00 +CPU= 33.889
  20:35:25 STEP 2872 H=2872:00 +CPU= 23.618
  20:35:47 STEP 2873 H=2873:00 +CPU= 22.345
  20:36:23 STEP 2874 H=2874:00 +CPU= 35.709
  20:36:45 STEP 2875 H=2875:00 +CPU= 22.862
  20:37:07 STEP 2876 H=2876:00 +CPU= 21.132
  20:37:40 STEP 2877 H=2877:00 +CPU= 32.957
  20:38:05 STEP 2878 H=2878:00 +CPU= 25.242
  20:38:29 STEP 2879 H=2879:00 +CPU= 24.249
  20:39:10 STEP 2880 H=2880:00 +CPU= 40.954
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_119.zip
Uploading the intermediate file: upload_file_119.zip
  20:39:34 STEP 2881 H=2881:00 +CPU= 23.468
Uploading trickle at timestep: 10368000
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0125
(argv4) batchid: 968
(argv5) wuid: 12184769
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/5
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_94efadbcf6e283bfae4ae785f5a2ae5a to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0125_1999050100_123_968_12184769.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0125_1999050100_123_968_12184769.zip
ic_ancil_file: ic_ancil_12184769
ifsdata_file: ifsdata_12184769
climate_data_file: clim_data_12184769
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_15979f9ba9ad98edd65fc7ea4df72e0e to: /var/lib/boinc-client_40000/slots/5/ic_ancil_12184769.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/5/ic_ancil_12184769.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_81790918821c74b33530f07c30103421 to: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12184769.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12184769.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12184769.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12184769.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/5/progress_file_12184769.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/5/progress_file_12184769.xml
-- Model is restarting --
Adjusting last_iter, 2881, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/5/progress_file_12184769.xml
last_cpu_time: 76504
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630
The child process has been launched with process id: 1463
Executing the command: /var/lib/boinc-client_40000/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1463:1463] [20230118:204956:1674074996:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:51:47 STEP 2880 H=2880:00 +CPU= 45.427
  20:52:11 STEP 2881 H=2881:00 +CPU= 23.834
  20:52:35 STEP 2882 H=2882:00 +CPU= 24.025
  20:53:13 STEP 2883 H=2883:00 +CPU= 38.661
  20:53:41 STEP 2884 H=2884:00 +CPU= 28.257
  20:54:04 STEP 2885 H=2885:00 +CPU= 22.867
  20:54:43 STEP 2886 H=2886:00 +CPU= 38.904
  20:55:06 STEP 2887 H=2887:00 +CPU= 22.925
  20:55:30 STEP 2888 H=2888:00 +CPU= 24.202
  20:56:05 STEP 2889 H=2889:00 +CPU= 34.986
  20:56:27 STEP 2890 H=2890:00 +CPU= 22.047
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0125
(argv4) batchid: 968
(argv5) wuid: 12184769
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client_40000/slots/5
Project directory is: /var/lib/boinc-client_40000/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_94efadbcf6e283bfae4ae785f5a2ae5a to: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0125_1999050100_123_968_12184769.zip
Unzipping the namelist zip file: /var/lib/boinc-client_40000/slots/5/oifs_43r3_ps_0125_1999050100_123_968_12184769.zip
ic_ancil_file: ic_ancil_12184769
ifsdata_file: ifsdata_12184769
climate_data_file: clim_data_12184769
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_15979f9ba9ad98edd65fc7ea4df72e0e to: /var/lib/boinc-client_40000/slots/5/ic_ancil_12184769.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client_40000/slots/5/ic_ancil_12184769.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_81790918821c74b33530f07c30103421 to: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12184769.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client_40000/slots/5/ifsdata/ifsdata_12184769.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12184769.zip
Unzipping the climate data zip file: /var/lib/boinc-client_40000/slots/5/159l_2/clim_data_12184769.zip
Checking for progress XML file: /var/lib/boinc-client_40000/slots/5/progress_file_12184769.xml
Opened progress file ok : /var/lib/boinc-client_40000/slots/5/progress_file_12184769.xml
-- Model is restarting --
Adjusting last_iter, 2890, to previous model restart step.
Creating progress file: /var/lib/boinc-client_40000/slots/5/progress_file_12184769.xml
last_cpu_time: 76832
upload_file_number: 120
last_iter: 2879
last_upload: 10368000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630
The child process has been launched with process id: 1450
Executing the command: /var/lib/boinc-client_40000/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:e1:1:1:1450:1450] [20230118:210618:1674075978:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:08:46 STEP 2880 H=2880:00 +CPU= 39.552
  21:09:08 STEP 2881 H=2881:00 +CPU= 21.798
  21:09:31 STEP 2882 H=2882:00 +CPU= 23.067
  21:10:05 STEP 2883 H=2883:00 +CPU= 33.780
  21:10:30 STEP 2884 H=2884:00 +CPU= 25.200
  21:10:54 STEP 2885 H=2885:00 +CPU= 24.448
  21:11:33 STEP 2886 H=2886:00 +CPU= 38.322
  21:11:58 STEP 2887 H=2887:00 +CPU= 25.208
  21:12:23 STEP 2888 H=2888:00 +CPU= 25.174
  21:13:02 STEP 2889 H=2889:00 +CPU= 38.860
  21:13:26 STEP 2890 H=2890:00 +CPU= 24.348
  21:13:49 STEP 2891 H=2891:00 +CPU= 22.406
  21:14:26 STEP 2892 H=2892:00 +CPU= 37.646
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002892
  21:14:46 STEP 2893 H=2893:00 +CPU= 19.536
  21:15:07 STEP 2894 H=2894:00 +CPU= 21.542
  21:15:40 STEP 2895 H=2895:00 +CPU= 32.360
  21:16:00 STEP 2896 H=2896:00 +CPU= 20.219
  21:16:19 STEP 2897 H=2897:00 +CPU= 19.274
  21:16:54 STEP 2898 H=2898:00 +CPU= 34.700
  21:17:14 STEP 2899 H=2899:00 +CPU= 20.089
  21:17:37 STEP 2900 H=2900:00 +CPU= 23.098
  21:18:13 STEP 2901 H=2901:00 +CPU= 35.767
  21:18:37 STEP 2902 H=2902:00 +CPU= 23.970
  21:19:00 STEP 2903 H=2903:00 +CPU= 22.587
  21:19:41 STEP 2904 H=2904:00 +CPU= 41.777
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_12184769/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client_40000/projects/climateprediction.net/oifs_43r3_ps_0125_1999050100_123_968_12184769_0_r849879630_120.zip
Uploading the intermediate file: upload_file_120.zip
  21:20:05 STEP 2905 H=2905:00 +CPU= 23.757
  21:20:30 STEP 2906 H=2906:00 +CPU= 24.354
  21:21:07 STEP 2907 H=2907:00 +CPU= 36.669
  21:21:31 STEP 2908 H=2908:00 +CPU= 24.352
  21:21:54 STEP 2909 H=2909:00 +CPU= 23.478
  21:22:33 STEP 2910 H=2910:00 +CPU= 38.173
  21:22:56 STEP 2911 H=2911:00 +CPU= 23.540
  21:23:20 STEP 2912 H=2912:00 +CPU= 24.013
  21:23:56 STEP 2913 H=2913:00 +CPU= 35.921
  21:24:23 STEP 2914 H=2914:00 +CPU= 26.920
  21:24:50 STEP 2915 H=2915:00 +CPU= 26.795
  21:25:35 STEP 2916 H=2916:00 +CPU= 44.884
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client_40000/slots/5/ICMUAhq0f+002916
  21:26:03 STEP 2917 H=2917:00 +CPU= 28.051
  21:26:30 STEP 2918 H=2918:00 +CPU= 27.061
  21:27:12 STEP 2919 H=2919:00 +CPU= 42.227
  21:27:40 STEP 2920 H=2920:00 +CPU= 27.812
  21:28:08 STEP 2921 H=2921:00 +CPU= 27.989
  21:28:54 STEP 2922 H=2922:00 +CPU= 45.462
  21:29:21 STEP 2923 H=2923:00 +CPU= 27.363
  21:29:51 STEP 2924 H=2924:00 +CPU= 29.996
  21:30:31 STEP 2925 H=2925:00 +CPU= 40.380
  21:30:57 STEP 2926 H=2926:00 +CPU= 25.592
  21:31:21 STEP 2927 H=2927:00 +CPU= 24.455

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:25:22 1538236 22270037 oifs_43r3_ps_0125_1999050100_123_968_12184769_0 1,728,000 10,574 0.0061
09 May 2023 16:25:16 1538236 22270037 oifs_43r3_ps_0125_1999050100_123_968_12184769_0 864,000 4,392 0.0051


©2024 climateprediction.net