climateprediction.net home page
Task 22279965

Task 22279965

Name oifs_43r3_ps_0498_2008050100_123_977_12194142_0
Workunit 12194142
Created 23 Dec 2022, 18:35:33 UTC
Sent 22 Jan 2023, 0:22:20 UTC
Report deadline 23 Mar 2023, 0:22:20 UTC
Received 31 Jan 2023, 8:15:02 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1526454
Run time 1 days 9 hours 55 min 20 sec
CPU time 20 hours 28 min 57 sec
Validate state Invalid
Credit 3,386.88
Device peak FLOPS 6.89 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 3,911.25 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,721.00 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
 /var/lib/boinc/slots/17/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_26.zip
Uploading the intermediate file: upload_file_26.zip
  14:25:22 STEP  649 H= 649:00 +CPU= 61.052
  14:26:50 STEP  650 H= 650:00 +CPU= 67.981
  14:29:04 STEP  651 H= 651:00 +CPU= 96.100
  14:30:24 STEP  652 H= 652:00 +CPU= 63.210
  14:31:55 STEP  653 H= 653:00 +CPU= 69.256
  14:34:25 STEP  654 H= 654:00 +CPU=100.134
  14:35:51 STEP  655 H= 655:00 +CPU= 63.963
  14:37:15 STEP  656 H= 656:00 +CPU= 66.778
  14:39:29 STEP  657 H= 657:00 +CPU= 93.880
  14:40:49 STEP  658 H= 658:00 +CPU= 61.195
  14:42:23 STEP  659 H= 659:00 +CPU= 67.503
  14:44:52 STEP  660 H= 660:00 +CPU=105.586
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000660
  14:46:24 STEP  661 H= 661:00 +CPU= 67.757
  14:47:47 STEP  662 H= 662:00 +CPU= 64.055
  14:50:03 STEP  663 H= 663:00 +CPU= 97.117
  14:51:35 STEP  664 H= 664:00 +CPU= 69.863
  14:52:56 STEP  665 H= 665:00 +CPU= 61.301
  14:55:09 STEP  666 H= 666:00 +CPU= 98.742
  14:56:31 STEP  667 H= 667:00 +CPU= 67.043
  14:57:56 STEP  668 H= 668:00 +CPU= 63.893
  14:59:56 STEP  669 H= 669:00 +CPU= 84.046
  15:01:08 STEP  670 H= 670:00 +CPU= 56.519
  15:02:24 STEP  671 H= 671:00 +CPU= 56.574
  15:04:37 STEP  672 H= 672:00 +CPU= 97.594
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_27.zip
Uploading the intermediate file: upload_file_27.zip
Uploading trickle at timestep: 2419200
  15:06:00 STEP  673 H= 673:00 +CPU= 57.698
  15:07:17 STEP  674 H= 674:00 +CPU= 56.957
  15:09:45 STEP  675 H= 675:00 +CPU= 91.512
  15:11:12 STEP  676 H= 676:00 +CPU= 62.245
  15:12:38 STEP  677 H= 677:00 +CPU= 58.897
  15:14:50 STEP  678 H= 678:00 +CPU= 94.695
  15:16:17 STEP  679 H= 679:00 +CPU= 60.986
  15:17:48 STEP  680 H= 680:00 +CPU= 63.475
  15:19:55 STEP  681 H= 681:00 +CPU= 94.101
  15:21:18 STEP  682 H= 682:00 +CPU= 58.780
  15:22:43 STEP  683 H= 683:00 +CPU= 60.590
  15:24:57 STEP  684 H= 684:00 +CPU= 99.187
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000684
  15:26:23 STEP  685 H= 685:00 +CPU= 61.157
  15:27:57 STEP  686 H= 686:00 +CPU= 68.262
  15:30:00 STEP  687 H= 687:00 +CPU= 88.146
  15:31:28 STEP  688 H= 688:00 +CPU= 61.687
  15:33:02 STEP  689 H= 689:00 +CPU= 69.036
  15:35:21 STEP  690 H= 690:00 +CPU= 94.338
  15:36:49 STEP  691 H= 691:00 +CPU= 63.343
  15:38:18 STEP  692 H= 692:00 +CPU= 68.080
  15:40:24 STEP  693 H= 693:00 +CPU= 87.208
  15:41:58 STEP  694 H= 694:00 +CPU= 68.976
  15:43:17 STEP  695 H= 695:00 +CPU= 59.876
  15:45:36 STEP  696 H= 696:00 +CPU=101.603
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_28.zip
Uploading the intermediate file: upload_file_28.zip
  15:47:04 STEP  697 H= 697:00 +CPU= 67.923
  15:48:10 STEP  698 H= 698:00 +CPU= 51.708
  15:50:46 STEP  699 H= 699:00 +CPU=101.093
  15:52:06 STEP  700 H= 700:00 +CPU= 64.064
  15:53:23 STEP  701 H= 701:00 +CPU= 54.999
  15:56:06 STEP  702 H= 702:00 +CPU=103.559
  15:57:30 STEP  703 H= 703:00 +CPU= 58.915
  15:58:56 STEP  704 H= 704:00 +CPU= 59.903
  16:01:01 STEP  705 H= 705:00 +CPU= 93.708
  16:02:30 STEP  706 H= 706:00 +CPU= 62.561
  16:03:54 STEP  707 H= 707:00 +CPU= 59.695
  16:06:14 STEP  708 H= 708:00 +CPU=105.570
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000708
  16:07:39 STEP  709 H= 709:00 +CPU= 59.356
  16:09:13 STEP  710 H= 710:00 +CPU= 66.895
  16:11:23 STEP  711 H= 711:00 +CPU= 93.650
  16:12:51 STEP  712 H= 712:00 +CPU= 62.468
  16:14:26 STEP  713 H= 713:00 +CPU= 67.620
  16:16:38 STEP  714 H= 714:00 +CPU= 94.098
  16:18:10 STEP  715 H= 715:00 +CPU= 64.501
  16:19:37 STEP  716 H= 716:00 +CPU= 64.049
  16:21:34 STEP  717 H= 717:00 +CPU= 85.355
  16:23:04 STEP  718 H= 718:00 +CPU= 65.699
  16:24:42 STEP  719 H= 719:00 +CPU= 70.140
  16:26:57 STEP  720 H= 720:00 +CPU= 98.431
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_29.zip
Uploading the intermediate file: upload_file_29.zip
  16:28:32 STEP  721 H= 721:00 +CPU= 68.002
  16:30:03 STEP  722 H= 722:00 +CPU= 63.370
  16:32:46 STEP  723 H= 723:00 +CPU= 99.258
  16:34:20 STEP  724 H= 724:00 +CPU= 67.156
  16:35:39 STEP  725 H= 725:00 +CPU= 55.439
  16:38:09 STEP  726 H= 726:00 +CPU= 99.110
  16:39:40 STEP  727 H= 727:00 +CPU= 64.307
  16:41:07 STEP  728 H= 728:00 +CPU= 60.366
  16:43:19 STEP  729 H= 729:00 +CPU= 97.274
  16:44:50 STEP  730 H= 730:00 +CPU= 61.849
  16:46:26 STEP  731 H= 731:00 +CPU= 65.446
  16:48:41 STEP  732 H= 732:00 +CPU= 98.565
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000732
  16:50:17 STEP  733 H= 733:00 +CPU= 61.021
  16:51:57 STEP  734 H= 734:00 +CPU= 69.175
  16:54:06 STEP  735 H= 735:00 +CPU= 86.056
  16:55:45 STEP  736 H= 736:00 +CPU= 68.108
  16:57:20 STEP  737 H= 737:00 +CPU= 67.887
  16:59:25 STEP  738 H= 738:00 +CPU= 84.385
  17:00:56 STEP  739 H= 739:00 +CPU= 62.473
  17:02:29 STEP  740 H= 740:00 +CPU= 63.268
  17:04:27 STEP  741 H= 741:00 +CPU= 79.398
  17:06:02 STEP  742 H= 742:00 +CPU= 65.093
  17:07:19 STEP  743 H= 743:00 +CPU= 55.525
  17:09:28 STEP  744 H= 744:00 +CPU= 95.475
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_30.zip
Uploading the intermediate file: upload_file_30.zip
  17:10:47 STEP  745 H= 745:00 +CPU= 60.725
  17:11:55 STEP  746 H= 746:00 +CPU= 50.673
  17:13:49 STEP  747 H= 747:00 +CPU= 81.854
  17:15:18 STEP  748 H= 748:00 +CPU= 62.827
  17:16:43 STEP  749 H= 749:00 +CPU= 55.629
  17:18:46 STEP  750 H= 750:00 +CPU= 89.996
  17:20:07 STEP  751 H= 751:00 +CPU= 57.664
  17:21:26 STEP  752 H= 752:00 +CPU= 54.664
  17:23:23 STEP  753 H= 753:00 +CPU= 88.315
  17:24:38 STEP  754 H= 754:00 +CPU= 57.618
  17:25:48 STEP  755 H= 755:00 +CPU= 51.716
  17:27:46 STEP  756 H= 756:00 +CPU= 90.501
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000756
  17:28:59 STEP  757 H= 757:00 +CPU= 55.933
  17:30:12 STEP  758 H= 758:00 +CPU= 52.876
  17:32:02 STEP  759 H= 759:00 +CPU= 83.808
  17:33:15 STEP  760 H= 760:00 +CPU= 56.253
  17:34:20 STEP  761 H= 761:00 +CPU= 48.893
  17:36:18 STEP  762 H= 762:00 +CPU= 88.049
  17:37:32 STEP  763 H= 763:00 +CPU= 56.999
  17:38:33 STEP  764 H= 764:00 +CPU= 46.885
  17:40:23 STEP  765 H= 765:00 +CPU= 84.544
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0498
(argv4) batchid: 977
(argv5) wuid: 12194142
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/17
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1f49e8ce6723d8d180b0fe9039361f3f to: /var/lib/boinc/slots/17/oifs_43r3_ps_0498_2008050100_123_977_12194142.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/17/oifs_43r3_ps_0498_2008050100_123_977_12194142.zip
ic_ancil_file: ic_ancil_12194142
ifsdata_file: ifsdata_12194142
climate_data_file: clim_data_12194142
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4dd70a2ec3c643c0c51bf52ed2be396e to: /var/lib/boinc/slots/17/ic_ancil_12194142.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/17/ic_ancil_12194142.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_dab1a3fd29a8180bfff6762899c18f45 to: /var/lib/boinc/slots/17/ifsdata/ifsdata_12194142.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/17/ifsdata/ifsdata_12194142.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/17/159l_2/clim_data_12194142.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/17/159l_2/clim_data_12194142.zip
Checking for progress XML file: /var/lib/boinc/slots/17/progress_file_12194142.xml
Opened progress file ok : /var/lib/boinc/slots/17/progress_file_12194142.xml
-- Model is restarting --
Adjusting last_iter, 766, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/17/progress_file_12194142.xml
last_cpu_time: 52096
upload_file_number: 31
last_iter: 743
last_upload: 2678400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604
The child process has been launched with process id: 4372
Executing the command: /var/lib/boinc/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:eikan.pc:1:1:4372:4372] [20230130:174410:1675093450:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:45:51 STEP  744 H= 744:00 +CPU= 75.230
  17:47:00 STEP  745 H= 745:00 +CPU= 58.393
  17:48:00 STEP  746 H= 746:00 +CPU= 55.514
  17:49:20 STEP  747 H= 747:00 +CPU= 77.126
  17:50:00 STEP  748 H= 748:00 +CPU= 37.117
  17:51:02 STEP  749 H= 749:00 +CPU= 55.059
  17:52:26 STEP  750 H= 750:00 +CPU= 77.947
  17:53:24 STEP  751 H= 751:00 +CPU= 54.394
  17:54:11 STEP  752 H= 752:00 +CPU= 45.932
  17:55:21 STEP  753 H= 753:00 +CPU= 63.793
  17:56:20 STEP  754 H= 754:00 +CPU= 55.110
  17:57:13 STEP  755 H= 755:00 +CPU= 51.599
  17:58:38 STEP  756 H= 756:00 +CPU= 81.443
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000756
  17:59:18 STEP  757 H= 757:00 +CPU= 36.534
  18:00:17 STEP  758 H= 758:00 +CPU= 54.697
  18:01:32 STEP  759 H= 759:00 +CPU= 74.054
  18:02:25 STEP  760 H= 760:00 +CPU= 52.214
  18:03:14 STEP  761 H= 761:00 +CPU= 48.467
  18:04:23 STEP  762 H= 762:00 +CPU= 61.817
  18:05:15 STEP  763 H= 763:00 +CPU= 50.893
  18:06:10 STEP  764 H= 764:00 +CPU= 54.300
  18:07:25 STEP  765 H= 765:00 +CPU= 73.880
  18:08:08 STEP  766 H= 766:00 +CPU= 42.351
  18:09:04 STEP  767 H= 767:00 +CPU= 51.462
  18:10:39 STEP  768 H= 768:00 +CPU= 92.254
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_31.zip
Uploading the intermediate file: upload_file_31.zip
  18:11:30 STEP  769 H= 769:00 +CPU= 50.166
  18:12:21 STEP  770 H= 770:00 +CPU= 49.779
  18:13:35 STEP  771 H= 771:00 +CPU= 66.042
  18:14:30 STEP  772 H= 772:00 +CPU= 53.481
  18:15:21 STEP  773 H= 773:00 +CPU= 50.214
  18:16:43 STEP  774 H= 774:00 +CPU= 79.002
  18:17:25 STEP  775 H= 775:00 +CPU= 40.092
  18:18:22 STEP  776 H= 776:00 +CPU= 50.693
  18:19:49 STEP  777 H= 777:00 +CPU= 80.371
  18:20:44 STEP  778 H= 778:00 +CPU= 53.508
  18:21:33 STEP  779 H= 779:00 +CPU= 47.410
  18:22:55 STEP  780 H= 780:00 +CPU= 72.747
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000780
  18:23:59 STEP  781 H= 781:00 +CPU= 56.007
  18:24:56 STEP  782 H= 782:00 +CPU= 55.086
  18:26:09 STEP  783 H= 783:00 +CPU= 71.753
  18:26:58 STEP  784 H= 784:00 +CPU= 44.638
  18:27:56 STEP  785 H= 785:00 +CPU= 52.544
  18:29:14 STEP  786 H= 786:00 +CPU= 75.548
  18:30:05 STEP  787 H= 787:00 +CPU= 49.460
  18:30:51 STEP  788 H= 788:00 +CPU= 45.878
  18:32:10 STEP  789 H= 789:00 +CPU= 69.295
  18:33:08 STEP  790 H= 790:00 +CPU= 53.424
  18:34:03 STEP  791 H= 791:00 +CPU= 53.158
  18:35:38 STEP  792 H= 792:00 +CPU= 88.065
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_32.zip
Uploading the intermediate file: upload_file_32.zip
  18:36:29 STEP  793 H= 793:00 +CPU= 44.856
  18:37:26 STEP  794 H= 794:00 +CPU= 54.953
  18:38:38 STEP  795 H= 795:00 +CPU= 71.584
  18:39:33 STEP  796 H= 796:00 +CPU= 52.628
  18:40:16 STEP  797 H= 797:00 +CPU= 42.355
  18:41:39 STEP  798 H= 798:00 +CPU= 74.311
  18:42:34 STEP  799 H= 799:00 +CPU= 53.780
  18:43:31 STEP  800 H= 800:00 +CPU= 56.221
  18:44:42 STEP  801 H= 801:00 +CPU= 69.220
  18:45:28 STEP  802 H= 802:00 +CPU= 45.149
  18:46:27 STEP  803 H= 803:00 +CPU= 53.799
  18:47:52 STEP  804 H= 804:00 +CPU= 82.197
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000804
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000804
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000804
  18:48:42 STEP  805 H= 805:00 +CPU= 48.851
  18:49:30 STEP  806 H= 806:00 +CPU= 45.029
  18:50:48 STEP  807 H= 807:00 +CPU= 70.983
  18:51:43 STEP  808 H= 808:00 +CPU= 52.780
  18:52:37 STEP  809 H= 809:00 +CPU= 53.318
  18:53:50 STEP  810 H= 810:00 +CPU= 70.947
  18:54:39 STEP  811 H= 811:00 +CPU= 46.481
  18:55:43 STEP  812 H= 812:00 +CPU= 54.163
  18:57:08 STEP  813 H= 813:00 +CPU= 79.374
  18:58:00 STEP  814 H= 814:00 +CPU= 51.084
  18:58:49 STEP  815 H= 815:00 +CPU= 45.669
  19:00:29 STEP  816 H= 816:00 +CPU= 90.055
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000816
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000816
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000804
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000804
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_33.zip
Uploading the intermediate file: upload_file_33.zip
  19:01:19 STEP  817 H= 817:00 +CPU= 48.697
  19:02:21 STEP  818 H= 818:00 +CPU= 54.980
  19:03:40 STEP  819 H= 819:00 +CPU= 67.196
  19:04:35 STEP  820 H= 820:00 +CPU= 49.832
  19:05:43 STEP  821 H= 821:00 +CPU= 58.298
  19:07:03 STEP  822 H= 822:00 +CPU= 73.799
  19:07:55 STEP  823 H= 823:00 +CPU= 45.898
  19:08:54 STEP  824 H= 824:00 +CPU= 52.810
  19:10:17 STEP  825 H= 825:00 +CPU= 77.852
  19:11:11 STEP  826 H= 826:00 +CPU= 50.103
  19:12:01 STEP  827 H= 827:00 +CPU= 43.573
  19:13:31 STEP  828 H= 828:00 +CPU= 80.848
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000828
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000828
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000828
  19:14:32 STEP  829 H= 829:00 +CPU= 54.916
  19:15:28 STEP  830 H= 830:00 +CPU= 52.391
  19:16:53 STEP  831 H= 831:00 +CPU= 69.912
  19:17:51 STEP  832 H= 832:00 +CPU= 52.927
  19:18:46 STEP  833 H= 833:00 +CPU= 51.690
  19:20:06 STEP  834 H= 834:00 +CPU= 75.271
  19:20:55 STEP  835 H= 835:00 +CPU= 44.871
  19:21:52 STEP  836 H= 836:00 +CPU= 50.393
  19:23:24 STEP  837 H= 837:00 +CPU= 79.771
  19:24:18 STEP  838 H= 838:00 +CPU= 50.200
  19:25:21 STEP  839 H= 839:00 +CPU= 52.942
  19:26:49 STEP  840 H= 840:00 +CPU= 82.721
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000840
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000840
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000816
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000828
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000828
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_34.zip
Uploading the intermediate file: upload_file_34.zip
  19:27:45 STEP  841 H= 841:00 +CPU= 52.417
  19:28:37 STEP  842 H= 842:00 +CPU= 50.913
  19:29:51 STEP  843 H= 843:00 +CPU= 69.075
  19:30:48 STEP  844 H= 844:00 +CPU= 50.795
  19:31:38 STEP  845 H= 845:00 +CPU= 48.596
  19:32:56 STEP  846 H= 846:00 +CPU= 75.218
  19:33:51 STEP  847 H= 847:00 +CPU= 51.615
  19:34:36 STEP  848 H= 848:00 +CPU= 44.037
  19:35:54 STEP  849 H= 849:00 +CPU= 72.826
  19:36:45 STEP  850 H= 850:00 +CPU= 49.023
  19:37:33 STEP  851 H= 851:00 +CPU= 47.312
  19:38:52 STEP  852 H= 852:00 +CPU= 75.373
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000852
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000852
  19:39:44 STEP  853 H= 853:00 +CPU= 49.696
  19:40:38 STEP  854 H= 854:00 +CPU= 53.187
  19:41:57 STEP  855 H= 855:00 +CPU= 75.831
  19:42:51 STEP  856 H= 856:00 +CPU= 51.906
  19:43:38 STEP  857 H= 857:00 +CPU= 45.050
  19:44:58 STEP  858 H= 858:00 +CPU= 75.990
  19:45:52 STEP  859 H= 859:00 +CPU= 51.223
  19:46:48 STEP  860 H= 860:00 +CPU= 54.673
  19:48:03 STEP  861 H= 861:00 +CPU= 70.131
  19:48:57 STEP  862 H= 862:00 +CPU= 49.980
  19:49:51 STEP  863 H= 863:00 +CPU= 52.467
  19:51:31 STEP  864 H= 864:00 +CPU= 93.486
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000864
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000864
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000840
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000852
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000852
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_35.zip
Uploading the intermediate file: upload_file_35.zip
  19:52:27 STEP  865 H= 865:00 +CPU= 53.053
  19:53:19 STEP  866 H= 866:00 +CPU= 47.169
  19:54:39 STEP  867 H= 867:00 +CPU= 75.691
  19:55:35 STEP  868 H= 868:00 +CPU= 53.370
  19:56:31 STEP  869 H= 869:00 +CPU= 53.912
  19:57:54 STEP  870 H= 870:00 +CPU= 74.568
  19:58:49 STEP  871 H= 871:00 +CPU= 52.300
  19:59:39 STEP  872 H= 872:00 +CPU= 48.126
  20:01:01 STEP  873 H= 873:00 +CPU= 76.787
  20:01:54 STEP  874 H= 874:00 +CPU= 49.878
  20:02:43 STEP  875 H= 875:00 +CPU= 46.341
  20:04:10 STEP  876 H= 876:00 +CPU= 81.440
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000876
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000876
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000876
  20:05:04 STEP  877 H= 877:00 +CPU= 51.323
  20:06:02 STEP  878 H= 878:00 +CPU= 52.391
  20:07:21 STEP  879 H= 879:00 +CPU= 72.060
  20:08:20 STEP  880 H= 880:00 +CPU= 55.390
  20:09:12 STEP  881 H= 881:00 +CPU= 49.405
  20:10:38 STEP  882 H= 882:00 +CPU= 79.648
  20:11:32 STEP  883 H= 883:00 +CPU= 47.487
  20:12:27 STEP  884 H= 884:00 +CPU= 52.535
  20:13:45 STEP  885 H= 885:00 +CPU= 74.141
  20:14:39 STEP  886 H= 886:00 +CPU= 51.343
  20:15:32 STEP  887 H= 887:00 +CPU= 49.764
  20:17:11 STEP  888 H= 888:00 +CPU= 90.815
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000888
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000888
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000864
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000876
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_36.zip
Uploading the intermediate file: upload_file_36.zip
  20:18:00 STEP  889 H= 889:00 +CPU= 47.112
  20:18:54 STEP  890 H= 890:00 +CPU= 52.443
  20:20:15 STEP  891 H= 891:00 +CPU= 73.445
  20:21:05 STEP  892 H= 892:00 +CPU= 48.806
  20:21:56 STEP  893 H= 893:00 +CPU= 49.873
  20:23:27 STEP  894 H= 894:00 +CPU= 79.436
  20:24:25 STEP  895 H= 895:00 +CPU= 53.031
  20:25:27 STEP  896 H= 896:00 +CPU= 52.499
  20:26:50 STEP  897 H= 897:00 +CPU= 77.200
  20:27:46 STEP  898 H= 898:00 +CPU= 49.707
  20:28:43 STEP  899 H= 899:00 +CPU= 54.147
  20:30:15 STEP  900 H= 900:00 +CPU= 81.753
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000900
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000900
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000900
  20:31:13 STEP  901 H= 901:00 +CPU= 52.153
  20:32:14 STEP  902 H= 902:00 +CPU= 51.806
  20:33:41 STEP  903 H= 903:00 +CPU= 78.839
  20:34:42 STEP  904 H= 904:00 +CPU= 49.963
  20:35:39 STEP  905 H= 905:00 +CPU= 51.430
  20:37:04 STEP  906 H= 906:00 +CPU= 73.348
  20:38:04 STEP  907 H= 907:00 +CPU= 53.998
  20:39:04 STEP  908 H= 908:00 +CPU= 51.350
  20:40:32 STEP  909 H= 909:00 +CPU= 78.526
  20:41:30 STEP  910 H= 910:00 +CPU= 50.880
  20:42:28 STEP  911 H= 911:00 +CPU= 52.855
  20:44:06 STEP  912 H= 912:00 +CPU= 86.618
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000912
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000912
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000888
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000900
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000900
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_37.zip
Uploading the intermediate file: upload_file_37.zip
  20:45:04 STEP  913 H= 913:00 +CPU= 54.225
  20:45:53 STEP  914 H= 914:00 +CPU= 46.651
  20:47:18 STEP  915 H= 915:00 +CPU= 77.562
  20:48:08 STEP  916 H= 916:00 +CPU= 47.710
  20:49:01 STEP  917 H= 917:00 +CPU= 49.622
  20:50:24 STEP  918 H= 918:00 +CPU= 75.631
  20:51:16 STEP  919 H= 919:00 +CPU= 49.751
  20:52:11 STEP  920 H= 920:00 +CPU= 51.486
  20:53:33 STEP  921 H= 921:00 +CPU= 73.735
  20:54:25 STEP  922 H= 922:00 +CPU= 48.722
  20:55:16 STEP  923 H= 923:00 +CPU= 46.628
  20:56:46 STEP  924 H= 924:00 +CPU= 82.848
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000924
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000924
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000924
  20:57:36 STEP  925 H= 925:00 +CPU= 46.363
  20:58:33 STEP  926 H= 926:00 +CPU= 51.742
  20:59:51 STEP  927 H= 927:00 +CPU= 70.386
  21:00:47 STEP  928 H= 928:00 +CPU= 52.516
  21:01:37 STEP  929 H= 929:00 +CPU= 45.436
  21:03:04 STEP  930 H= 930:00 +CPU= 78.671
  21:03:55 STEP  931 H= 931:00 +CPU= 46.392
  21:04:54 STEP  932 H= 932:00 +CPU= 52.245
  21:06:07 STEP  933 H= 933:00 +CPU= 67.743
  21:07:10 STEP  934 H= 934:00 +CPU= 54.833
  21:08:04 STEP  935 H= 935:00 +CPU= 49.934
  21:09:42 STEP  936 H= 936:00 +CPU= 86.384
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000936
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000936
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000912
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000924
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000924
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_38.zip
Uploading the intermediate file: upload_file_38.zip
  21:10:30 STEP  937 H= 937:00 +CPU= 43.566
  21:11:26 STEP  938 H= 938:00 +CPU= 49.246
  21:12:45 STEP  939 H= 939:00 +CPU= 73.002
  21:13:42 STEP  940 H= 940:00 +CPU= 51.173
  21:14:33 STEP  941 H= 941:00 +CPU= 46.253
  21:15:55 STEP  942 H= 942:00 +CPU= 71.218
  21:16:56 STEP  943 H= 943:00 +CPU= 54.533
  21:17:50 STEP  944 H= 944:00 +CPU= 47.345
  21:19:07 STEP  945 H= 945:00 +CPU= 70.987
  21:20:08 STEP  946 H= 946:00 +CPU= 51.042
  21:21:04 STEP  947 H= 947:00 +CPU= 51.703
  21:22:31 STEP  948 H= 948:00 +CPU= 79.324
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000948
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000948
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000948
  21:23:28 STEP  949 H= 949:00 +CPU= 51.120
  21:24:20 STEP  950 H= 950:00 +CPU= 46.280
  21:25:49 STEP  951 H= 951:00 +CPU= 77.463
  21:26:44 STEP  952 H= 952:00 +CPU= 50.073
  21:27:37 STEP  953 H= 953:00 +CPU= 46.859
  21:29:01 STEP  954 H= 954:00 +CPU= 73.209
  21:29:58 STEP  955 H= 955:00 +CPU= 51.676
  21:30:53 STEP  956 H= 956:00 +CPU= 49.232
  21:32:16 STEP  957 H= 957:00 +CPU= 73.878
  21:33:12 STEP  958 H= 958:00 +CPU= 48.375
  21:34:10 STEP  959 H= 959:00 +CPU= 53.313
  21:35:50 STEP  960 H= 960:00 +CPU= 90.542
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000960
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000960
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000936
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000948
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000948
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_39.zip
Uploading the intermediate file: upload_file_39.zip
  21:36:40 STEP  961 H= 961:00 +CPU= 46.370
  21:37:30 STEP  962 H= 962:00 +CPU= 45.760
  21:38:59 STEP  963 H= 963:00 +CPU= 79.986
  21:39:55 STEP  964 H= 964:00 +CPU= 51.564
  21:40:50 STEP  965 H= 965:00 +CPU= 49.673
  21:42:11 STEP  966 H= 966:00 +CPU= 72.361
  21:43:16 STEP  967 H= 967:00 +CPU= 57.193
  21:44:16 STEP  968 H= 968:00 +CPU= 50.035
  21:45:44 STEP  969 H= 969:00 +CPU= 76.253
  21:46:46 STEP  970 H= 970:00 +CPU= 49.583
  21:47:49 STEP  971 H= 971:00 +CPU= 56.403
  21:49:34 STEP  972 H= 972:00 +CPU= 89.148
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000972
  21:50:26 STEP  973 H= 973:00 +CPU= 46.234
  21:51:18 STEP  974 H= 974:00 +CPU= 47.148
  21:52:48 STEP  975 H= 975:00 +CPU= 79.037
  21:53:49 STEP  976 H= 976:00 +CPU= 54.253
  21:54:49 STEP  977 H= 977:00 +CPU= 52.497
  21:56:08 STEP  978 H= 978:00 +CPU= 71.610
  21:57:12 STEP  979 H= 979:00 +CPU= 57.309
  21:58:10 STEP  980 H= 980:00 +CPU= 50.822
  21:59:30 STEP  981 H= 981:00 +CPU= 72.482
  22:00:36 STEP  982 H= 982:00 +CPU= 52.538
  22:01:46 STEP  983 H= 983:00 +CPU= 57.040
  22:03:34 STEP  984 H= 984:00 +CPU= 91.698
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_40.zip
Uploading the intermediate file: upload_file_40.zip
Uploading trickle at timestep: 3542400
  22:04:38 STEP  985 H= 985:00 +CPU= 52.128
  22:05:39 STEP  986 H= 986:00 +CPU= 50.543
  22:07:09 STEP  987 H= 987:00 +CPU= 77.518
  22:08:13 STEP  988 H= 988:00 +CPU= 57.812
  22:09:06 STEP  989 H= 989:00 +CPU= 48.366
  22:10:22 STEP  990 H= 990:00 +CPU= 69.086
  22:11:20 STEP  991 H= 991:00 +CPU= 53.184
  22:12:16 STEP  992 H= 992:00 +CPU= 51.610
  22:13:36 STEP  993 H= 993:00 +CPU= 73.222
  22:14:33 STEP  994 H= 994:00 +CPU= 49.686
  22:15:29 STEP  995 H= 995:00 +CPU= 52.699
  22:17:02 STEP  996 H= 996:00 +CPU= 87.757
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+000996
  22:17:51 STEP  997 H= 997:00 +CPU= 47.375
  22:18:36 STEP  998 H= 998:00 +CPU= 42.708
  22:19:52 STEP  999 H= 999:00 +CPU= 71.225
  22:20:43 STEP 1000 H=1000:00 +CPU= 48.835
  22:21:37 STEP 1001 H=1001:00 +CPU= 52.015
  22:22:57 STEP 1002 H=1002:00 +CPU= 74.420
  22:23:49 STEP 1003 H=1003:00 +CPU= 46.599
  22:24:46 STEP 1004 H=1004:00 +CPU= 52.226
  22:26:08 STEP 1005 H=1005:00 +CPU= 78.019
  22:27:02 STEP 1006 H=1006:00 +CPU= 51.066
  22:27:53 STEP 1007 H=1007:00 +CPU= 45.119
  22:29:29 STEP 1008 H=1008:00 +CPU= 89.278
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_41.zip
Uploading the intermediate file: upload_file_41.zip
  22:30:34 STEP 1009 H=1009:00 +CPU= 58.303
  22:31:28 STEP 1010 H=1010:00 +CPU= 50.082
  22:32:46 STEP 1011 H=1011:00 +CPU= 68.292
  22:33:47 STEP 1012 H=1012:00 +CPU= 55.092
  22:34:43 STEP 1013 H=1013:00 +CPU= 52.844
  22:36:04 STEP 1014 H=1014:00 +CPU= 76.656
  22:36:58 STEP 1015 H=1015:00 +CPU= 48.753
  22:37:54 STEP 1016 H=1016:00 +CPU= 50.483
  22:39:13 STEP 1017 H=1017:00 +CPU= 75.285
  22:40:03 STEP 1018 H=1018:00 +CPU= 47.970
  22:40:51 STEP 1019 H=1019:00 +CPU= 46.956
  22:42:10 STEP 1020 H=1020:00 +CPU= 70.372
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001020
  22:43:04 STEP 1021 H=1021:00 +CPU= 51.195
  22:43:54 STEP 1022 H=1022:00 +CPU= 48.430
  22:45:05 STEP 1023 H=1023:00 +CPU= 67.398
  22:45:51 STEP 1024 H=1024:00 +CPU= 44.601
  22:46:36 STEP 1025 H=1025:00 +CPU= 43.561
  22:47:49 STEP 1026 H=1026:00 +CPU= 70.108
  22:48:41 STEP 1027 H=1027:00 +CPU= 50.572
  22:49:32 STEP 1028 H=1028:00 +CPU= 49.615
  22:50:38 STEP 1029 H=1029:00 +CPU= 63.025
  22:51:32 STEP 1030 H=1030:00 +CPU= 50.033
  22:52:24 STEP 1031 H=1031:00 +CPU= 50.193
  22:53:59 STEP 1032 H=1032:00 +CPU= 87.256
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_42.zip
Uploading the intermediate file: upload_file_42.zip
  22:54:50 STEP 1033 H=1033:00 +CPU= 45.816
  22:55:34 STEP 1034 H=1034:00 +CPU= 41.947
  22:56:53 STEP 1035 H=1035:00 +CPU= 71.824
  22:57:41 STEP 1036 H=1036:00 +CPU= 46.368
  22:58:30 STEP 1037 H=1037:00 +CPU= 47.508
  22:59:44 STEP 1038 H=1038:00 +CPU= 66.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:00:52 STEP 1039 H=1039:00 +CPU= 34.341
  07:01:58 STEP 1040 H=1040:00 +CPU= 27.678
  07:03:52 STEP 1041 H=1041:00 +CPU= 44.123
  07:05:03 STEP 1042 H=1042:00 +CPU= 28.002
  07:06:25 STEP 1043 H=1043:00 +CPU= 32.269
  07:08:25 STEP 1044 H=1044:00 +CPU= 47.456
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001044
  07:09:45 STEP 1045 H=1045:00 +CPU= 32.323
  07:11:00 STEP 1046 H=1046:00 +CPU= 29.012
  07:13:01 STEP 1047 H=1047:00 +CPU= 46.631
  07:14:18 STEP 1048 H=1048:00 +CPU= 28.880
  07:15:36 STEP 1049 H=1049:00 +CPU= 31.015
  07:17:31 STEP 1050 H=1050:00 +CPU= 44.110
  07:18:50 STEP 1051 H=1051:00 +CPU= 30.311
  07:20:08 STEP 1052 H=1052:00 +CPU= 32.581
  07:21:59 STEP 1053 H=1053:00 +CPU= 41.893
  07:23:14 STEP 1054 H=1054:00 +CPU= 30.087
  07:24:30 STEP 1055 H=1055:00 +CPU= 29.036
  07:26:43 STEP 1056 H=1056:00 +CPU= 53.923
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_43.zip
Uploading the intermediate file: upload_file_43.zip
  07:28:02 STEP 1057 H=1057:00 +CPU= 28.960
  07:29:22 STEP 1058 H=1058:00 +CPU= 30.225
  07:31:21 STEP 1059 H=1059:00 +CPU= 44.858
  07:33:04 STEP 1060 H=1060:00 +CPU= 32.709
  07:34:22 STEP 1061 H=1061:00 +CPU= 31.170
  07:36:12 STEP 1062 H=1062:00 +CPU= 43.086
  07:37:34 STEP 1063 H=1063:00 +CPU= 31.751
  07:38:46 STEP 1064 H=1064:00 +CPU= 27.426
  07:40:44 STEP 1065 H=1065:00 +CPU= 45.062
  07:41:52 STEP 1066 H=1066:00 +CPU= 27.374
  07:43:33 STEP 1067 H=1067:00 +CPU= 33.670
  07:45:36 STEP 1068 H=1068:00 +CPU= 47.499
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001068
  07:47:23 STEP 1069 H=1069:00 +CPU= 34.047
  07:48:35 STEP 1070 H=1070:00 +CPU= 28.423
  07:50:29 STEP 1071 H=1071:00 +CPU= 44.657
  07:51:49 STEP 1072 H=1072:00 +CPU= 30.784
  07:53:35 STEP 1073 H=1073:00 +CPU= 33.471
  07:55:25 STEP 1074 H=1074:00 +CPU= 43.151
  07:56:43 STEP 1075 H=1075:00 +CPU= 27.950
  07:57:57 STEP 1076 H=1076:00 +CPU= 27.120
  07:59:46 STEP 1077 H=1077:00 +CPU= 39.983
  08:01:26 STEP 1078 H=1078:00 +CPU= 28.836
  08:02:28 STEP 1079 H=1079:00 +CPU= 25.213
  08:04:25 STEP 1080 H=1080:00 +CPU= 41.972
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_44.zip
Uploading the intermediate file: upload_file_44.zip
  08:05:37 STEP 1081 H=1081:00 +CPU= 26.393
  08:06:48 STEP 1082 H=1082:00 +CPU= 25.368
  08:08:36 STEP 1083 H=1083:00 +CPU= 37.675
  08:09:37 STEP 1084 H=1084:00 +CPU= 23.969
  08:10:41 STEP 1085 H=1085:00 +CPU= 24.146
  08:12:26 STEP 1086 H=1086:00 +CPU= 38.551
  08:13:32 STEP 1087 H=1087:00 +CPU= 24.293
  08:14:44 STEP 1088 H=1088:00 +CPU= 26.537
  08:16:27 STEP 1089 H=1089:00 +CPU= 36.527
  08:17:43 STEP 1090 H=1090:00 +CPU= 27.275
  08:18:56 STEP 1091 H=1091:00 +CPU= 25.454
  08:20:42 STEP 1092 H=1092:00 +CPU= 39.278
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001092
  08:22:25 STEP 1093 H=1093:00 +CPU= 29.691
  08:23:33 STEP 1094 H=1094:00 +CPU= 25.320
  08:25:22 STEP 1095 H=1095:00 +CPU= 37.566
  08:26:34 STEP 1096 H=1096:00 +CPU= 25.864
  08:28:07 STEP 1097 H=1097:00 +CPU= 27.498
  08:29:56 STEP 1098 H=1098:00 +CPU= 38.664
  08:30:59 STEP 1099 H=1099:00 +CPU= 23.887
  08:32:31 STEP 1100 H=1100:00 +CPU= 27.440
  08:34:15 STEP 1101 H=1101:00 +CPU= 38.341
  08:35:26 STEP 1102 H=1102:00 +CPU= 25.668
  08:36:25 STEP 1103 H=1103:00 +CPU= 23.503
  08:38:32 STEP 1104 H=1104:00 +CPU= 44.350
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_45.zip
Uploading the intermediate file: upload_file_45.zip
  08:39:38 STEP 1105 H=1105:00 +CPU= 23.955
  08:40:48 STEP 1106 H=1106:00 +CPU= 25.546
  08:42:33 STEP 1107 H=1107:00 +CPU= 36.891
  08:43:33 STEP 1108 H=1108:00 +CPU= 23.621
  08:44:45 STEP 1109 H=1109:00 +CPU= 25.472
  08:46:39 STEP 1110 H=1110:00 +CPU= 39.143
  08:47:46 STEP 1111 H=1111:00 +CPU= 25.138
  08:49:21 STEP 1112 H=1112:00 +CPU= 27.790
  08:51:07 STEP 1113 H=1113:00 +CPU= 38.320
  08:52:19 STEP 1114 H=1114:00 +CPU= 25.803
  08:53:56 STEP 1115 H=1115:00 +CPU= 28.644
  08:55:42 STEP 1116 H=1116:00 +CPU= 40.008
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001116
  08:56:49 STEP 1117 H=1117:00 +CPU= 24.346
  08:57:57 STEP 1118 H=1118:00 +CPU= 25.166
  08:59:43 STEP 1119 H=1119:00 +CPU= 38.395
  09:01:05 STEP 1120 H=1120:00 +CPU= 25.369
  09:02:12 STEP 1121 H=1121:00 +CPU= 25.337
  09:03:57 STEP 1122 H=1122:00 +CPU= 38.678
  09:05:00 STEP 1123 H=1123:00 +CPU= 24.178
  09:06:00 STEP 1124 H=1124:00 +CPU= 23.878
  09:07:50 STEP 1125 H=1125:00 +CPU= 38.011
  09:08:50 STEP 1126 H=1126:00 +CPU= 24.194
  09:09:50 STEP 1127 H=1127:00 +CPU= 23.309
  09:11:35 STEP 1128 H=1128:00 +CPU= 42.376
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_46.zip
Uploading the intermediate file: upload_file_46.zip
  09:12:28 STEP 1129 H=1129:00 +CPU= 20.832
  09:13:23 STEP 1130 H=1130:00 +CPU= 23.321
  09:14:59 STEP 1131 H=1131:00 +CPU= 35.424
  09:16:04 STEP 1132 H=1132:00 +CPU= 25.731
  09:17:05 STEP 1133 H=1133:00 +CPU= 24.360
  09:18:49 STEP 1134 H=1134:00 +CPU= 37.773
  09:19:51 STEP 1135 H=1135:00 +CPU= 24.472
  09:20:51 STEP 1136 H=1136:00 +CPU= 24.067
  09:22:31 STEP 1137 H=1137:00 +CPU= 38.124
  09:23:32 STEP 1138 H=1138:00 +CPU= 24.805
  09:24:55 STEP 1139 H=1139:00 +CPU= 28.620
  09:26:32 STEP 1140 H=1140:00 +CPU= 39.187
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001140
  09:27:24 STEP 1141 H=1141:00 +CPU= 23.413
  09:28:23 STEP 1142 H=1142:00 +CPU= 25.156
  09:29:54 STEP 1143 H=1143:00 +CPU= 36.321
  09:30:53 STEP 1144 H=1144:00 +CPU= 25.551
  09:31:49 STEP 1145 H=1145:00 +CPU= 23.785
  09:33:27 STEP 1146 H=1146:00 +CPU= 39.824
  09:34:28 STEP 1147 H=1147:00 +CPU= 24.903
  09:35:32 STEP 1148 H=1148:00 +CPU= 25.861
  09:37:06 STEP 1149 H=1149:00 +CPU= 39.261
  09:38:10 STEP 1150 H=1150:00 +CPU= 26.662
  09:39:18 STEP 1151 H=1151:00 +CPU= 27.328
  09:41:02 STEP 1152 H=1152:00 +CPU= 43.358
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0498_2008050100_123_977_12194142_0_r625960604_47.zip
Uploading the intermediate file: upload_file_47.zip
  09:42:24 STEP 1153 H=1153:00 +CPU= 29.020
  09:43:16 STEP 1154 H=1154:00 +CPU= 23.293
  09:44:51 STEP 1155 H=1155:00 +CPU= 38.112
  09:45:43 STEP 1156 H=1156:00 +CPU= 21.930
  09:46:40 STEP 1157 H=1157:00 +CPU= 21.496
  09:48:11 STEP 1158 H=1158:00 +CPU= 35.668
  09:49:05 STEP 1159 H=1159:00 +CPU= 21.900
  09:50:09 STEP 1160 H=1160:00 +CPU= 24.459
  09:51:38 STEP 1161 H=1161:00 +CPU= 34.118
  09:52:28 STEP 1162 H=1162:00 +CPU= 20.561
  09:53:25 STEP 1163 H=1163:00 +CPU= 22.794
  09:54:54 STEP 1164 H=1164:00 +CPU= 36.230
Moving to projects directory: /var/lib/boinc/slots/17/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/17/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc/slots/17/ICMUAhq0f+001164
  09:55:47 STEP 1165 H=1165:00 +CPU= 22.047
  09:56:47 STEP 1166 H=1166:00 +CPU= 23.479
  09:58:18 STEP 1167 H=1167:00 +CPU= 34.156
  09:59:10 STEP 1168 H=1168:00 +CPU= 22.219
  10:00:04 STEP 1169 H=1169:00 +CPU= 21.417
  10:01:30 STEP 1170 H=1170:00 +CPU= 33.690
  10:02:18 STEP 1171 H=1171:00 +CPU= 20.400
  10:03:13 STEP 1172 H=1172:00 +CPU= 21.735
  10:04:37 STEP 1173 H=1173:00 +CPU= 33.537
  10:05:24 STEP 1174 H=1174:00 +CPU= 20.240
  10:06:20 STEP 1175 H=1175:00 +CPU= 22.899
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0498
(argv4) batchid: 977
(argv5) wuid: 12194142
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/17
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1f49e8ce6723d8d180b0fe9039361f3f to: /var/lib/boinc/slots/17/oifs_43r3_ps_0498_2008050100_123_977_12194142.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/17/oifs_43r3_ps_0498_2008050100_123_977_12194142.zip
ic_ancil_file: ic_ancil_12194142
ifsdata_file: ifsdata_12194142
climate_data_file: clim_data_12194142
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4dd70a2ec3c643c0c51bf52ed2be396e to: /var/lib/boinc/slots/17/ic_ancil_12194142.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/17/ic_ancil_12194142.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2008050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0498
(argv4) batchid: 977
(argv5) wuid: 12194142
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/17
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12194142
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_1f49e8ce6723d8d180b0fe9039361f3f to: /var/lib/boinc/slots/17/oifs_43r3_ps_0498_2008050100_123_977_12194142.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/17/oifs_43r3_ps_0498_2008050100_123_977_12194142.zip
ic_ancil_file: ic_ancil_12194142
ifsdata_file: ifsdata_12194142
climate_data_file: clim_data_12194142
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: &#147; to: /var/lib/boinc/slots/17/ic_ancil_12194142.zip
..Copying the IC ancils to the working directory failed

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:45:41 1526454 22279965 oifs_43r3_ps_0498_2008050100_123_977_12194142_0 3,542,400 66,398 0.0187
09 May 2023 16:45:41 1526454 22279965 oifs_43r3_ps_0498_2008050100_123_977_12194142_0 2,419,200 45,441 0.0188
09 May 2023 16:42:29 1526454 22279965 oifs_43r3_ps_0498_2008050100_123_977_12194142_0 864,000 14,761 0.0171


©2024 climateprediction.net