climateprediction.net home page
Task 22442702

Task 22442702

Name oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0
Workunit 12289124
Created 12 Jun 2024, 21:07:36 UTC
Sent 13 Jun 2024, 21:55:19 UTC
Report deadline 12 Aug 2024, 21:55:19 UTC
Received 24 Jun 2024, 11:34:13 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1551031
Run time 15 hours 32 min 52 sec
CPU time 11 hours 39 min 49 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 4.04 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,025.53 MB
Peak swap size 5,981.12 MB
Peak disk usage 1,977.83 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
et/oifs_43r3_bl_12289124/ICMGGh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0_r956614650_6.zip
Uploading the intermediate file: upload_file_6.zip
  23:40:34 STEP  673 H= 168:15 +CPU= 41.042
  23:41:39 STEP  674 H= 168:30 +CPU= 40.343
  23:42:27 STEP  675 H= 168:45 +CPU= 34.224
  23:43:32 STEP  676 H= 169:00 +CPU= 40.693
  23:44:36 STEP  677 H= 169:15 +CPU= 40.945
  23:45:42 STEP  678 H= 169:30 +CPU= 41.132
  23:46:37 STEP  679 H= 169:45 +CPU= 36.165
  23:47:30 STEP  680 H= 170:00 +CPU= 38.039
  23:48:28 STEP  681 H= 170:15 +CPU= 40.996
  23:49:25 STEP  682 H= 170:30 +CPU= 40.834
  23:50:21 STEP  683 H= 170:45 +CPU= 41.011
  23:51:18 STEP  684 H= 171:00 +CPU= 44.631
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000684
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000684
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000684
  23:52:06 STEP  685 H= 171:15 +CPU= 37.320
  23:52:59 STEP  686 H= 171:30 +CPU= 40.723
  23:53:53 STEP  687 H= 171:45 +CPU= 40.768
  23:54:47 STEP  688 H= 172:00 +CPU= 41.783
  23:55:36 STEP  689 H= 172:15 +CPU= 37.451
  23:56:20 STEP  690 H= 172:30 +CPU= 34.489
  23:57:11 STEP  691 H= 172:45 +CPU= 40.780
  23:58:05 STEP  692 H= 173:00 +CPU= 41.607
  23:58:58 STEP  693 H= 173:15 +CPU= 40.794
  23:59:47 STEP  694 H= 173:30 +CPU= 36.351
  00:00:32 STEP  695 H= 173:45 +CPU= 34.770
  00:01:36 STEP  696 H= 174:00 +CPU= 50.918
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000696
  00:02:32 STEP  697 H= 174:15 +CPU= 40.967
  00:03:24 STEP  698 H= 174:30 +CPU= 40.579
  00:04:06 STEP  699 H= 174:45 +CPU= 34.589
  00:04:50 STEP  700 H= 175:00 +CPU= 36.657
  00:05:40 STEP  701 H= 175:15 +CPU= 40.717
  00:06:31 STEP  702 H= 175:30 +CPU= 40.738
  00:07:20 STEP  703 H= 175:45 +CPU= 40.339
  00:08:02 STEP  704 H= 176:00 +CPU= 34.869
  00:08:48 STEP  705 H= 176:15 +CPU= 37.692
  00:09:38 STEP  706 H= 176:30 +CPU= 40.749
  00:10:28 STEP  707 H= 176:45 +CPU= 40.679
  00:11:25 STEP  708 H= 177:00 +CPU= 46.712
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000708
  00:12:07 STEP  709 H= 177:15 +CPU= 34.592
  00:12:56 STEP  710 H= 177:30 +CPU= 39.591
  00:13:47 STEP  711 H= 177:45 +CPU= 40.564
  00:14:39 STEP  712 H= 178:00 +CPU= 41.521
  00:15:27 STEP  713 H= 178:15 +CPU= 38.907
  00:16:09 STEP  714 H= 178:30 +CPU= 34.207
  00:16:56 STEP  715 H= 178:45 +CPU= 38.155
  00:17:47 STEP  716 H= 179:00 +CPU= 41.406
  00:18:38 STEP  717 H= 179:15 +CPU= 40.533
  00:19:25 STEP  718 H= 179:30 +CPU= 38.166
  00:20:08 STEP  719 H= 179:45 +CPU= 34.408
  00:21:16 STEP  720 H= 180:00 +CPU= 52.864
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000720
  00:22:05 STEP  721 H= 180:15 +CPU= 40.600
  00:22:53 STEP  722 H= 180:30 +CPU= 40.709
  00:23:36 STEP  723 H= 180:45 +CPU= 35.872
  00:24:16 STEP  724 H= 181:00 +CPU= 34.719
  00:25:04 STEP  725 H= 181:15 +CPU= 39.985
  00:25:51 STEP  726 H= 181:30 +CPU= 40.447
  00:26:38 STEP  727 H= 181:45 +CPU= 40.569
  00:27:22 STEP  728 H= 182:00 +CPU= 35.918
  00:28:05 STEP  729 H= 182:15 +CPU= 34.901
  00:28:59 STEP  730 H= 182:30 +CPU= 41.272
  00:29:52 STEP  731 H= 182:45 +CPU= 41.006
  00:30:57 STEP  732 H= 183:00 +CPU= 51.060
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000732
  00:31:49 STEP  733 H= 183:15 +CPU= 41.139
  00:32:39 STEP  734 H= 183:30 +CPU= 40.905
  00:33:30 STEP  735 H= 183:45 +CPU= 40.839
  00:34:21 STEP  736 H= 184:00 +CPU= 41.716
  00:35:12 STEP  737 H= 184:15 +CPU= 40.961
  00:36:03 STEP  738 H= 184:30 +CPU= 41.281
  00:36:52 STEP  739 H= 184:45 +CPU= 40.838
  00:37:43 STEP  740 H= 185:00 +CPU= 42.161
  00:38:31 STEP  741 H= 185:15 +CPU= 40.832
  00:39:19 STEP  742 H= 185:30 +CPU= 40.991
  00:40:07 STEP  743 H= 185:45 +CPU= 41.040
  00:41:07 STEP  744 H= 186:00 +CPU= 51.164
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000744
  00:41:57 STEP  745 H= 186:15 +CPU= 41.328
  00:42:45 STEP  746 H= 186:30 +CPU= 41.072
  00:43:32 STEP  747 H= 186:45 +CPU= 40.817
  00:44:20 STEP  748 H= 187:00 +CPU= 41.534
  00:45:07 STEP  749 H= 187:15 +CPU= 40.706
  00:45:54 STEP  750 H= 187:30 +CPU= 40.831
  00:46:41 STEP  751 H= 187:45 +CPU= 40.731
  00:47:30 STEP  752 H= 188:00 +CPU= 41.780
  00:48:17 STEP  753 H= 188:15 +CPU= 41.005
  00:49:04 STEP  754 H= 188:30 +CPU= 40.848
  00:49:51 STEP  755 H= 188:45 +CPU= 40.858
  00:50:51 STEP  756 H= 189:00 +CPU= 51.131
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000756
  00:51:38 STEP  757 H= 189:15 +CPU= 41.068
  00:52:25 STEP  758 H= 189:30 +CPU= 41.099
  00:53:11 STEP  759 H= 189:45 +CPU= 41.039
  00:53:58 STEP  760 H= 190:00 +CPU= 41.728
  00:54:43 STEP  761 H= 190:15 +CPU= 40.703
  00:55:29 STEP  762 H= 190:30 +CPU= 40.688
  00:56:15 STEP  763 H= 190:45 +CPU= 41.093
  00:57:02 STEP  764 H= 191:00 +CPU= 41.932
  00:57:48 STEP  765 H= 191:15 +CPU= 41.034
  00:58:34 STEP  766 H= 191:30 +CPU= 40.797
  00:59:19 STEP  767 H= 191:45 +CPU= 40.628
  01:00:21 STEP  768 H= 192:00 +CPU= 55.053
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0_r956614650_7.zip
Uploading the intermediate file: upload_file_7.zip
  01:01:07 STEP  769 H= 192:15 +CPU= 40.773
  01:01:53 STEP  770 H= 192:30 +CPU= 40.914
  01:02:40 STEP  771 H= 192:45 +CPU= 41.160
  01:03:28 STEP  772 H= 193:00 +CPU= 42.086
  01:04:14 STEP  773 H= 193:15 +CPU= 40.637
  01:04:59 STEP  774 H= 193:30 +CPU= 40.824
  01:05:43 STEP  775 H= 193:45 +CPU= 41.218
  01:06:29 STEP  776 H= 194:00 +CPU= 41.562
  01:07:13 STEP  777 H= 194:15 +CPU= 40.981
  01:07:58 STEP  778 H= 194:30 +CPU= 40.727
  01:08:41 STEP  779 H= 194:45 +CPU= 40.264
  01:09:35 STEP  780 H= 195:00 +CPU= 50.403
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000780
  01:10:19 STEP  781 H= 195:15 +CPU= 40.338
  01:11:03 STEP  782 H= 195:30 +CPU= 41.087
  01:11:47 STEP  783 H= 195:45 +CPU= 40.498
  01:12:33 STEP  784 H= 196:00 +CPU= 42.315
  01:13:17 STEP  785 H= 196:15 +CPU= 40.697
  01:14:01 STEP  786 H= 196:30 +CPU= 40.721
  01:14:45 STEP  787 H= 196:45 +CPU= 41.125
  01:15:30 STEP  788 H= 197:00 +CPU= 41.701
  01:16:13 STEP  789 H= 197:15 +CPU= 40.546
  01:16:57 STEP  790 H= 197:30 +CPU= 41.005
  01:17:41 STEP  791 H= 197:45 +CPU= 40.808
  01:18:35 STEP  792 H= 198:00 +CPU= 50.297
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000792
  01:19:19 STEP  793 H= 198:15 +CPU= 40.967
  01:20:03 STEP  794 H= 198:30 +CPU= 40.688
  01:20:46 STEP  795 H= 198:45 +CPU= 40.582
  01:21:31 STEP  796 H= 199:00 +CPU= 41.673
  01:22:15 STEP  797 H= 199:15 +CPU= 40.599
  01:22:58 STEP  798 H= 199:30 +CPU= 40.772
  01:23:42 STEP  799 H= 199:45 +CPU= 40.650
  01:24:27 STEP  800 H= 200:00 +CPU= 42.304
  01:25:11 STEP  801 H= 200:15 +CPU= 41.240
  01:25:55 STEP  802 H= 200:30 +CPU= 40.649
  01:26:39 STEP  803 H= 200:45 +CPU= 40.739
  01:27:33 STEP  804 H= 201:00 +CPU= 50.607
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000804
  01:28:17 STEP  805 H= 201:15 +CPU= 41.280
  01:29:02 STEP  806 H= 201:30 +CPU= 41.395
  01:29:46 STEP  807 H= 201:45 +CPU= 41.195
  01:30:31 STEP  808 H= 202:00 +CPU= 42.145
  01:31:15 STEP  809 H= 202:15 +CPU= 40.835
  01:31:59 STEP  810 H= 202:30 +CPU= 40.905
  01:32:43 STEP  811 H= 202:45 +CPU= 41.047
  01:33:28 STEP  812 H= 203:00 +CPU= 42.156
  01:34:12 STEP  813 H= 203:15 +CPU= 41.575
  01:34:57 STEP  814 H= 203:30 +CPU= 41.463
  01:35:42 STEP  815 H= 203:45 +CPU= 41.475
  01:36:43 STEP  816 H= 204:00 +CPU= 56.117
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000816
  01:37:28 STEP  817 H= 204:15 +CPU= 41.764
  01:38:13 STEP  818 H= 204:30 +CPU= 41.306
  01:38:57 STEP  819 H= 204:45 +CPU= 41.137
  01:39:42 STEP  820 H= 205:00 +CPU= 41.682
  01:40:26 STEP  821 H= 205:15 +CPU= 40.854
  01:41:10 STEP  822 H= 205:30 +CPU= 41.612
  01:41:55 STEP  823 H= 205:45 +CPU= 41.111
  01:42:40 STEP  824 H= 206:00 +CPU= 41.803
  01:43:24 STEP  825 H= 206:15 +CPU= 41.075
  01:44:09 STEP  826 H= 206:30 +CPU= 41.204
  01:44:53 STEP  827 H= 206:45 +CPU= 41.277
  01:45:49 STEP  828 H= 207:00 +CPU= 51.617
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000828
  01:46:33 STEP  829 H= 207:15 +CPU= 41.467
  01:47:19 STEP  830 H= 207:30 +CPU= 41.587
  01:48:04 STEP  831 H= 207:45 +CPU= 41.942
  01:48:49 STEP  832 H= 208:00 +CPU= 42.335
  01:49:32 STEP  833 H= 208:15 +CPU= 41.358
  01:50:18 STEP  834 H= 208:30 +CPU= 42.317
  01:51:01 STEP  835 H= 208:45 +CPU= 40.920
  01:51:45 STEP  836 H= 209:00 +CPU= 42.176
  01:52:28 STEP  837 H= 209:15 +CPU= 41.039
  01:53:10 STEP  838 H= 209:30 +CPU= 40.784
  01:53:53 STEP  839 H= 209:45 +CPU= 41.124
  01:54:45 STEP  840 H= 210:00 +CPU= 50.180
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000840
  01:55:28 STEP  841 H= 210:15 +CPU= 41.678
  01:56:11 STEP  842 H= 210:30 +CPU= 41.773
  01:56:53 STEP  843 H= 210:45 +CPU= 40.744
  01:57:37 STEP  844 H= 211:00 +CPU= 42.252
  01:58:20 STEP  845 H= 211:15 +CPU= 40.767
  01:59:02 STEP  846 H= 211:30 +CPU= 41.073
  01:59:45 STEP  847 H= 211:45 +CPU= 41.090
  02:00:29 STEP  848 H= 212:00 +CPU= 42.277
  02:01:11 STEP  849 H= 212:15 +CPU= 41.024
  02:01:54 STEP  850 H= 212:30 +CPU= 41.022
  02:02:36 STEP  851 H= 212:45 +CPU= 40.322
  02:03:28 STEP  852 H= 213:00 +CPU= 50.204
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000852
  02:04:10 STEP  853 H= 213:15 +CPU= 40.648
  02:04:52 STEP  854 H= 213:30 +CPU= 40.007
  02:05:34 STEP  855 H= 213:45 +CPU= 40.711
  02:06:17 STEP  856 H= 214:00 +CPU= 40.908
  02:07:00 STEP  857 H= 214:15 +CPU= 41.552
  02:07:43 STEP  858 H= 214:30 +CPU= 42.099
  02:08:26 STEP  859 H= 214:45 +CPU= 41.196
  02:09:10 STEP  860 H= 215:00 +CPU= 41.962
  02:09:53 STEP  861 H= 215:15 +CPU= 41.650
  02:10:36 STEP  862 H= 215:30 +CPU= 40.931
  02:11:19 STEP  863 H= 215:45 +CPU= 42.202
  02:12:18 STEP  864 H= 216:00 +CPU= 55.967
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0_r956614650_8.zip
Uploading the intermediate file: upload_file_8.zip
  02:13:03 STEP  865 H= 216:15 +CPU= 42.474
  02:13:46 STEP  866 H= 216:30 +CPU= 41.034
  02:14:30 STEP  867 H= 216:45 +CPU= 41.866
  02:15:12 STEP  868 H= 217:00 +CPU= 40.835
  02:15:54 STEP  869 H= 217:15 +CPU= 40.259
  02:16:37 STEP  870 H= 217:30 +CPU= 41.230
  02:17:20 STEP  871 H= 217:45 +CPU= 41.377
  02:18:04 STEP  872 H= 218:00 +CPU= 42.727
  02:18:46 STEP  873 H= 218:15 +CPU= 40.375
  02:19:31 STEP  874 H= 218:30 +CPU= 41.928
  02:20:14 STEP  875 H= 218:45 +CPU= 41.770
  02:21:08 STEP  876 H= 219:00 +CPU= 51.390
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000876
  02:21:50 STEP  877 H= 219:15 +CPU= 41.250
  02:22:34 STEP  878 H= 219:30 +CPU= 42.183
  02:23:16 STEP  879 H= 219:45 +CPU= 40.488
  02:24:00 STEP  880 H= 220:00 +CPU= 41.792
  02:24:43 STEP  881 H= 220:15 +CPU= 41.964
  02:25:26 STEP  882 H= 220:30 +CPU= 40.817
  02:26:08 STEP  883 H= 220:45 +CPU= 41.174
  02:26:52 STEP  884 H= 221:00 +CPU= 41.883
  02:27:35 STEP  885 H= 221:15 +CPU= 41.594
  02:28:18 STEP  886 H= 221:30 +CPU= 40.859
  02:29:00 STEP  887 H= 221:45 +CPU= 40.557
  02:29:52 STEP  888 H= 222:00 +CPU= 50.134
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000888
  02:30:34 STEP  889 H= 222:15 +CPU= 40.743
  02:31:17 STEP  890 H= 222:30 +CPU= 40.931
  02:32:01 STEP  891 H= 222:45 +CPU= 42.388
  02:32:44 STEP  892 H= 223:00 +CPU= 41.896
  02:33:26 STEP  893 H= 223:15 +CPU= 40.562
  02:34:09 STEP  894 H= 223:30 +CPU= 40.837
  02:34:51 STEP  895 H= 223:45 +CPU= 40.574
  02:35:35 STEP  896 H= 224:00 +CPU= 43.052
  02:36:19 STEP  897 H= 224:15 +CPU= 41.695
  02:37:02 STEP  898 H= 224:30 +CPU= 41.786
  02:37:46 STEP  899 H= 224:45 +CPU= 41.662
  02:38:40 STEP  900 H= 225:00 +CPU= 52.080
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000900
  02:39:23 STEP  901 H= 225:15 +CPU= 41.963
  02:40:07 STEP  902 H= 225:30 +CPU= 42.173
  02:40:50 STEP  903 H= 225:45 +CPU= 41.497
  02:41:34 STEP  904 H= 226:00 +CPU= 42.166
  02:42:17 STEP  905 H= 226:15 +CPU= 41.276
  02:43:00 STEP  906 H= 226:30 +CPU= 41.538
  02:43:44 STEP  907 H= 226:45 +CPU= 41.373
  02:44:28 STEP  908 H= 227:00 +CPU= 42.266
  02:45:13 STEP  909 H= 227:15 +CPU= 41.914
  02:45:56 STEP  910 H= 227:30 +CPU= 41.509
  02:46:39 STEP  911 H= 227:45 +CPU= 41.418
  02:47:36 STEP  912 H= 228:00 +CPU= 55.009
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000912
  02:48:20 STEP  913 H= 228:15 +CPU= 41.532
  02:49:04 STEP  914 H= 228:30 +CPU= 42.384
  02:49:48 STEP  915 H= 228:45 +CPU= 42.313
  02:50:31 STEP  916 H= 229:00 +CPU= 41.712
  02:51:15 STEP  917 H= 229:15 +CPU= 42.042
  02:51:59 STEP  918 H= 229:30 +CPU= 42.218
  02:52:42 STEP  919 H= 229:45 +CPU= 41.874
  02:53:27 STEP  920 H= 230:00 +CPU= 42.899
  02:54:11 STEP  921 H= 230:15 +CPU= 41.913
  02:54:55 STEP  922 H= 230:30 +CPU= 42.043
  02:55:38 STEP  923 H= 230:45 +CPU= 41.374
  02:56:32 STEP  924 H= 231:00 +CPU= 52.380
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000924
  02:57:16 STEP  925 H= 231:15 +CPU= 42.098
  02:58:00 STEP  926 H= 231:30 +CPU= 41.944
  02:58:43 STEP  927 H= 231:45 +CPU= 41.867
  02:59:27 STEP  928 H= 232:00 +CPU= 42.198
  03:00:09 STEP  929 H= 232:15 +CPU= 40.637
  03:00:53 STEP  930 H= 232:30 +CPU= 41.329
  03:01:36 STEP  931 H= 232:45 +CPU= 41.374
  03:02:19 STEP  932 H= 233:00 +CPU= 41.950
  03:03:03 STEP  933 H= 233:15 +CPU= 42.434
  03:03:47 STEP  934 H= 233:30 +CPU= 42.472
  03:04:32 STEP  935 H= 233:45 +CPU= 42.601
  03:05:25 STEP  936 H= 234:00 +CPU= 51.408
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000936
  03:06:08 STEP  937 H= 234:15 +CPU= 40.845
  03:06:51 STEP  938 H= 234:30 +CPU= 41.268
  03:07:34 STEP  939 H= 234:45 +CPU= 41.011
  03:08:18 STEP  940 H= 235:00 +CPU= 42.604
  03:09:02 STEP  941 H= 235:15 +CPU= 41.712
  03:09:44 STEP  942 H= 235:30 +CPU= 40.857
  03:10:27 STEP  943 H= 235:45 +CPU= 41.533
  03:11:10 STEP  944 H= 236:00 +CPU= 41.508
  03:11:53 STEP  945 H= 236:15 +CPU= 41.080
  03:12:37 STEP  946 H= 236:30 +CPU= 42.015
  03:13:19 STEP  947 H= 236:45 +CPU= 40.660
  03:14:12 STEP  948 H= 237:00 +CPU= 50.802
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000948
  03:14:56 STEP  949 H= 237:15 +CPU= 42.382
  03:15:40 STEP  950 H= 237:30 +CPU= 42.814
  03:16:23 STEP  951 H= 237:45 +CPU= 41.624
  03:17:07 STEP  952 H= 238:00 +CPU= 42.446
  03:17:50 STEP  953 H= 238:15 +CPU= 41.264
  03:18:33 STEP  954 H= 238:30 +CPU= 41.605
  03:19:15 STEP  955 H= 238:45 +CPU= 40.697
  03:19:58 STEP  956 H= 239:00 +CPU= 40.892
  03:20:42 STEP  957 H= 239:15 +CPU= 41.571
  03:21:24 STEP  958 H= 239:30 +CPU= 41.016
  03:22:08 STEP  959 H= 239:45 +CPU= 41.820
  03:23:05 STEP  960 H= 240:00 +CPU= 55.442
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0_r956614650_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  03:23:50 STEP  961 H= 240:15 +CPU= 42.069
  03:24:34 STEP  962 H= 240:30 +CPU= 42.280
  03:25:18 STEP  963 H= 240:45 +CPU= 42.007
  03:26:02 STEP  964 H= 241:00 +CPU= 42.374
  03:26:45 STEP  965 H= 241:15 +CPU= 41.597
  03:27:28 STEP  966 H= 241:30 +CPU= 41.786
  03:28:12 STEP  967 H= 241:45 +CPU= 42.141
  03:28:56 STEP  968 H= 242:00 +CPU= 42.425
  03:29:40 STEP  969 H= 242:15 +CPU= 41.724
  03:30:23 STEP  970 H= 242:30 +CPU= 41.563
  03:31:06 STEP  971 H= 242:45 +CPU= 41.926
  03:31:59 STEP  972 H= 243:00 +CPU= 51.387
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000972
  03:32:41 STEP  973 H= 243:15 +CPU= 40.710
  03:33:22 STEP  974 H= 243:30 +CPU= 39.725
  03:34:04 STEP  975 H= 243:45 +CPU= 41.110
  03:34:47 STEP  976 H= 244:00 +CPU= 41.671
  03:35:28 STEP  977 H= 244:15 +CPU= 40.791
  03:36:10 STEP  978 H= 244:30 +CPU= 40.480
  03:36:51 STEP  979 H= 244:45 +CPU= 39.959
  03:37:33 STEP  980 H= 245:00 +CPU= 41.667
  03:38:14 STEP  981 H= 245:15 +CPU= 39.668
  03:38:50 STEP  982 H= 245:30 +CPU= 35.864
  03:39:24 STEP  983 H= 245:45 +CPU= 32.653
  03:40:13 STEP  984 H= 246:00 +CPU= 48.393
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000984
  03:40:56 STEP  985 H= 246:15 +CPU= 42.314
  03:41:39 STEP  986 H= 246:30 +CPU= 41.333
  03:42:13 STEP  987 H= 246:45 +CPU= 33.871
  03:42:48 STEP  988 H= 247:00 +CPU= 34.635
  03:43:29 STEP  989 H= 247:15 +CPU= 39.745
  03:44:12 STEP  990 H= 247:30 +CPU= 41.832
  03:44:49 STEP  991 H= 247:45 +CPU= 36.526
  03:45:24 STEP  992 H= 248:00 +CPU= 34.712
  03:45:59 STEP  993 H= 248:15 +CPU= 33.876
  03:46:37 STEP  994 H= 248:30 +CPU= 37.624
  03:47:20 STEP  995 H= 248:45 +CPU= 42.310
  03:48:07 STEP  996 H= 249:00 +CPU= 45.760
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+000996
  03:48:43 STEP  997 H= 249:15 +CPU= 35.577
  03:49:18 STEP  998 H= 249:30 +CPU= 34.033
  03:49:51 STEP  999 H= 249:45 +CPU= 32.121
  03:50:32 STEP 1000 H= 250:00 +CPU= 40.286
  03:51:12 STEP 1001 H= 250:15 +CPU= 39.122
  03:51:47 STEP 1002 H= 250:30 +CPU= 34.467
  03:52:23 STEP 1003 H= 250:45 +CPU= 34.679
  03:52:59 STEP 1004 H= 251:00 +CPU= 35.233
  03:53:36 STEP 1005 H= 251:15 +CPU= 36.862
  03:54:17 STEP 1006 H= 251:30 +CPU= 40.090
  03:54:54 STEP 1007 H= 251:45 +CPU= 35.891
  03:55:42 STEP 1008 H= 252:00 +CPU= 47.016
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+001008
  03:56:15 STEP 1009 H= 252:15 +CPU= 32.935
  03:56:57 STEP 1010 H= 252:30 +CPU= 40.713
  03:57:39 STEP 1011 H= 252:45 +CPU= 41.242
  03:58:15 STEP 1012 H= 253:00 +CPU= 35.178
  03:58:49 STEP 1013 H= 253:15 +CPU= 33.823
  03:59:25 STEP 1014 H= 253:30 +CPU= 34.639
  04:00:02 STEP 1015 H= 253:45 +CPU= 36.647
  04:00:44 STEP 1016 H= 254:00 +CPU= 40.794
  04:01:23 STEP 1017 H= 254:15 +CPU= 38.405
  04:01:57 STEP 1018 H= 254:30 +CPU= 33.761
  04:02:30 STEP 1019 H= 254:45 +CPU= 31.904
  04:03:09 STEP 1020 H= 255:00 +CPU= 38.391
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+001020
  04:03:46 STEP 1021 H= 255:15 +CPU= 36.420
  04:04:26 STEP 1022 H= 255:30 +CPU= 39.107
  04:05:03 STEP 1023 H= 255:45 +CPU= 35.780
  04:05:39 STEP 1024 H= 256:00 +CPU= 35.487
  04:06:14 STEP 1025 H= 256:15 +CPU= 33.978
  04:06:48 STEP 1026 H= 256:30 +CPU= 33.954
  04:07:27 STEP 1027 H= 256:45 +CPU= 37.467
  04:08:07 STEP 1028 H= 257:00 +CPU= 39.501
  04:08:43 STEP 1029 H= 257:15 +CPU= 35.290
  04:09:17 STEP 1030 H= 257:30 +CPU= 33.529
  04:09:53 STEP 1031 H= 257:45 +CPU= 35.581
  04:10:37 STEP 1032 H= 258:00 +CPU= 42.961
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+001032
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+001032
  04:11:19 STEP 1033 H= 258:15 +CPU= 40.892
  04:11:58 STEP 1034 H= 258:30 +CPU= 37.991
  04:12:31 STEP 1035 H= 258:45 +CPU= 32.992
  04:13:07 STEP 1036 H= 259:00 +CPU= 35.601
  04:13:41 STEP 1037 H= 259:15 +CPU= 33.550
  04:14:22 STEP 1038 H= 259:30 +CPU= 39.237
  04:15:03 STEP 1039 H= 259:45 +CPU= 40.218
  04:15:40 STEP 1040 H= 260:00 +CPU= 36.188
  04:16:14 STEP 1041 H= 260:15 +CPU= 33.713
  04:16:49 STEP 1042 H= 260:30 +CPU= 34.410
  04:17:24 STEP 1043 H= 260:45 +CPU= 34.290
  04:18:07 STEP 1044 H= 261:00 +CPU= 42.226
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+001044
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+001044
  04:18:48 STEP 1045 H= 261:15 +CPU= 39.652
  04:19:28 STEP 1046 H= 261:30 +CPU= 39.653
  04:20:02 STEP 1047 H= 261:45 +CPU= 33.306
  04:20:36 STEP 1048 H= 262:00 +CPU= 33.733
  04:21:12 STEP 1049 H= 262:15 +CPU= 35.123
  04:21:47 STEP 1050 H= 262:30 +CPU= 34.568
  04:22:24 STEP 1051 H= 262:45 +CPU= 35.670
  04:23:06 STEP 1052 H= 263:00 +CPU= 41.495
  04:23:45 STEP 1053 H= 263:15 +CPU= 37.668
  04:24:18 STEP 1054 H= 263:30 +CPU= 32.876
  04:24:52 STEP 1055 H= 263:45 +CPU= 33.145
  04:25:37 STEP 1056 H= 264:00 +CPU= 43.807
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMGGh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMSHh7zg+001056
Moving to projects directory: /var/lib/boinc-client/slots/32/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMGGh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMSHh7zg+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12289124/ICMUAh7zg+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0_r956614650_10.zip
Uploading the intermediate file: upload_file_10.zip
  04:26:12 STEP 1057 H= 264:15 +CPU= 34.327
  04:26:51 STEP 1058 H= 264:30 +CPU= 38.447
  04:27:33 STEP 1059 H= 264:45 +CPU= 41.303
  04:28:09 STEP 1060 H= 265:00 +CPU= 35.384
  04:28:44 STEP 1061 H= 265:15 +CPU= 33.959
  04:29:18 STEP 1062 H= 265:30 +CPU= 33.736
  04:29:52 STEP 1063 H= 265:45 +CPU= 33.090
  04:30:31 STEP 1064 H= 266:00 +CPU= 38.161
  04:31:12 STEP 1065 H= 266:15 +CPU= 39.535
  04:31:46 STEP 1066 H= 266:30 +CPU= 33.484
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5026MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5026 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043210:1719225130:49329.941] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/32/oifs_43r3_model.exe' : sigcontextptr=0x7fff295c7310
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/32/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.943] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/32/oifs_43r3_model.exe' : sigcontextptr=0x7fff295c7220
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/32/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.970] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:P5810:1:1:2877479:2877479] [20240624:043211:1719225131:49330.970] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

 NSTEP =  1056 STEPO   F00000000
 CREATE RESTART FILE ./srf00110000.0001                                                                                                                                                                                                                                              
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./srf00110000.0001 MODE=w
 IO-STREAM CLOSED - ./srf00110000.0001
 MPI-TASK:   1 - ************ BYTES IN     46 RECORDS TRANSFERRED IN    2.6251 SECONDS******** Mbytes/s, TOTAL TIME=    3.2659(80.4%)
 CREATE RESTART CONTROL FILE ./rcf                                                                                                                   
 WRRESF: DELETED PREVIOUS RESTART FILE OK: ./srf00101200.0001
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1056 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001056 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001056
 MPI-TASK:   1 -    295432308 BYTES IN      6 RECORDS TRANSFERRED IN    0.0004 SECONDS738580.8 Mbytes/s, TOTAL TIME=    0.0133( 3.0%)
 NSTEP =  1056 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001056 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001056
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0098 SECONDS 72183.6 Mbytes/s, TOTAL TIME=    0.3713( 2.6%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001056 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001056
 MPI-TASK:   1 -   4726916928 BYTES IN    125 RECORDS TRANSFERRED IN    0.0065 SECONDS727218.0 Mbytes/s, TOTAL TIME=    0.2850( 2.3%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001056 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001056
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.0 Mbytes/s, TOTAL TIME=    0.0048( 4.2%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001056 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001056
 MPI-TASK:   1 -    286852579 BYTES IN      4 RECORDS TRANSFERRED IN    0.0003 SECONDS956175.3 Mbytes/s, TOTAL TIME=    0.0079( 3.8%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1056 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1056 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001056 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001056
 MPI-TASK:   1 -    295432308 BYTES IN      7 RECORDS TRANSFERRED IN    0.0003 SECONDS984774.3 Mbytes/s, TOTAL TIME=    0.0107( 2.8%)
 04:25:37 STEP 1056 H= 264:00 +CPU= 43.807
 NSTEP =  1057 STEPO   0AAA00AAA
 04:26:12 STEP 1057 H= 264:15 +CPU= 34.327
 NSTEP =  1058 STEPO   0AAA00AAA
 04:26:51 STEP 1058 H= 264:30 +CPU= 38.447
 NSTEP =  1059 STEPO   0AAA00AAA
 04:27:33 STEP 1059 H= 264:45 +CPU= 41.303
ISTEP=    1060 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1060 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1060 STEPO   0AAA00AAA
 04:28:09 STEP 1060 H= 265:00 +CPU= 35.384
 NSTEP =  1061 STEPO   0AAA00AAA
 04:28:44 STEP 1061 H= 265:15 +CPU= 33.959
 NSTEP =  1062 STEPO   0AAA00AAA
 04:29:18 STEP 1062 H= 265:30 +CPU= 33.736
 NSTEP =  1063 STEPO   0AAA00AAA
 04:29:52 STEP 1063 H= 265:45 +CPU= 33.090
 NSTEP =  1064 STEPO   0AAA00AAA
 04:30:31 STEP 1064 H= 266:00 +CPU= 38.161
ISTEP=    1065 RSOLINC= 1361.4482 RI0= 1365.0000 ZI0= 1361.4482 RIP0= 1361.4482
ISTEP=    1065 ZI0= 1361.4482 ZSEASON= 1.000000000 REA=149597870000.00 RDEASO=149597870000.00
 NSTEP =  1065 STEPO   0AAA00AAA
 04:31:12 STEP 1065 H= 266:15 +CPU= 39.535
 NSTEP =  1066 STEPO   0AAA00AAA
 04:31:46 STEP 1066 H= 266:30 +CPU= 33.484
 NSTEP =  1067 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 04:28:09 0AAA00AAA STEPO     1061    35.384   35.384   36.073    695:49    818:09 0.20039818503169E-04       2GB       0MB
 04:28:44 0AAA00AAA STEPO     1062    33.959   33.959   34.612    696:23    818:43 0.20039818503169E-04       2GB       0MB
 04:29:18 0AAA00AAA STEPO     1063    33.736   33.736   34.388    696:57    819:18 0.20039818503169E-04       2GB       0MB
 04:29:52 0AAA00AAA STEPO     1064    33.090   33.090   33.706    697:30    819:52 0.20039818503169E-04       2GB       0MB
 04:30:31 0AAA00AAA STEPO     1065    38.161   38.161   39.021    698:08    820:31 0.20039818503169E-04       2GB       0MB
 04:31:12 0AAA00AAA STEPO     1066    39.535   39.535   40.515    698:48    821:11 0.20039818503169E-04       2GB       0MB
 04:31:46 0AAA00AAA STEPO     1067    33.484   33.484   34.139    699:21    821:45 0.20039818503169E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1056",
 CTIME   = "00110000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,        1044, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/32/progress_file_12289124.xml

<running_values>
  <last_cpu_time>41988.240000</last_cpu_time>
  <upload_file_number>11</upload_file_number>
  <last_iter>1067</last_iter>
  <last_upload>950400</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
24 Jun 2024 10:22:33 1551031 22442702 oifs_43r3_bl_a0m5_2016092300_20_1018_12289124_0 864,000 37,941 0.0439


©2024 climateprediction.net