climateprediction.net home page
Task 22448142

Task 22448142

Name oifs_43r3_bl_a4s5_2016092300_20_1018_12294524_0
Workunit 12294524
Created 13 Jun 2024, 4:58:54 UTC
Sent 13 Jun 2024, 23:56:45 UTC
Report deadline 12 Aug 2024, 23:56:45 UTC
Received 26 Jun 2024, 0:35:29 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1551031
Run time 12 hours 48 min 49 sec
CPU time 11 hours 16 min 19 sec
Validate state Invalid
Credit 1,318.46
Device peak FLOPS 4.04 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,024.84 MB
Peak swap size 5,981.06 MB
Peak disk usage 1,348.92 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ing to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000612
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000612
  12:53:58 STEP  613 H= 153:15 +CPU= 35.928
  12:54:37 STEP  614 H= 153:30 +CPU= 37.968
  12:55:16 STEP  615 H= 153:45 +CPU= 38.794
  12:55:55 STEP  616 H= 154:00 +CPU= 37.972
  12:56:35 STEP  617 H= 154:15 +CPU= 39.228
  12:57:17 STEP  618 H= 154:30 +CPU= 40.647
  12:57:56 STEP  619 H= 154:45 +CPU= 37.794
  12:58:37 STEP  620 H= 155:00 +CPU= 40.467
  12:59:17 STEP  621 H= 155:15 +CPU= 38.546
  12:59:58 STEP  622 H= 155:30 +CPU= 40.474
  13:00:37 STEP  623 H= 155:45 +CPU= 38.671
  13:01:32 STEP  624 H= 156:00 +CPU= 52.995
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000624
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000624
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000624
  13:02:12 STEP  625 H= 156:15 +CPU= 39.711
  13:02:51 STEP  626 H= 156:30 +CPU= 38.094
  13:03:30 STEP  627 H= 156:45 +CPU= 38.157
  13:04:10 STEP  628 H= 157:00 +CPU= 38.536
  13:04:50 STEP  629 H= 157:15 +CPU= 39.094
  13:05:29 STEP  630 H= 157:30 +CPU= 38.250
  13:06:08 STEP  631 H= 157:45 +CPU= 38.513
  13:06:50 STEP  632 H= 158:00 +CPU= 40.173
  13:07:30 STEP  633 H= 158:15 +CPU= 39.610
  13:08:10 STEP  634 H= 158:30 +CPU= 39.051
  13:08:49 STEP  635 H= 158:45 +CPU= 38.224
  13:09:39 STEP  636 H= 159:00 +CPU= 48.962
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000636
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000636
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000636
  13:10:19 STEP  637 H= 159:15 +CPU= 39.093
  13:11:00 STEP  638 H= 159:30 +CPU= 40.293
  13:11:40 STEP  639 H= 159:45 +CPU= 38.530
  13:12:19 STEP  640 H= 160:00 +CPU= 38.539
  13:12:58 STEP  641 H= 160:15 +CPU= 37.937
  13:13:38 STEP  642 H= 160:30 +CPU= 38.705
  13:14:18 STEP  643 H= 160:45 +CPU= 39.556
  13:15:00 STEP  644 H= 161:00 +CPU= 40.992
  13:15:39 STEP  645 H= 161:15 +CPU= 38.274
  13:16:16 STEP  646 H= 161:30 +CPU= 35.995
  13:16:56 STEP  647 H= 161:45 +CPU= 39.283
  13:17:47 STEP  648 H= 162:00 +CPU= 49.291
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000648
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000648
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000648
  13:18:26 STEP  649 H= 162:15 +CPU= 37.942
  13:19:06 STEP  650 H= 162:30 +CPU= 39.199
  13:19:46 STEP  651 H= 162:45 +CPU= 39.282
  13:20:26 STEP  652 H= 163:00 +CPU= 39.324
  13:21:05 STEP  653 H= 163:15 +CPU= 38.205
  13:21:46 STEP  654 H= 163:30 +CPU= 39.599
  13:22:23 STEP  655 H= 163:45 +CPU= 36.726
  13:23:03 STEP  656 H= 164:00 +CPU= 38.972
  13:23:42 STEP  657 H= 164:15 +CPU= 38.569
  13:24:24 STEP  658 H= 164:30 +CPU= 40.356
  13:25:02 STEP  659 H= 164:45 +CPU= 37.636
  13:25:49 STEP  660 H= 165:00 +CPU= 46.268
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000660
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000660
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000660
  13:26:30 STEP  661 H= 165:15 +CPU= 39.513
  13:27:08 STEP  662 H= 165:30 +CPU= 37.359
  13:27:48 STEP  663 H= 165:45 +CPU= 39.095
  13:28:28 STEP  664 H= 166:00 +CPU= 38.789
  13:29:08 STEP  665 H= 166:15 +CPU= 39.519
  13:29:47 STEP  666 H= 166:30 +CPU= 37.968
  13:30:27 STEP  667 H= 166:45 +CPU= 39.402
  13:31:06 STEP  668 H= 167:00 +CPU= 38.169
  13:31:47 STEP  669 H= 167:15 +CPU= 39.461
  13:32:27 STEP  670 H= 167:30 +CPU= 39.480
  13:33:06 STEP  671 H= 167:45 +CPU= 38.215
  13:33:59 STEP  672 H= 168:00 +CPU= 51.423
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4s5_2016092300_20_1018_12294524_0_r859296012_6.zip
Uploading the intermediate file: upload_file_6.zip
  13:34:38 STEP  673 H= 168:15 +CPU= 38.429
  13:35:19 STEP  674 H= 168:30 +CPU= 39.474
  13:35:59 STEP  675 H= 168:45 +CPU= 39.335
  13:36:40 STEP  676 H= 169:00 +CPU= 40.690
  13:37:22 STEP  677 H= 169:15 +CPU= 40.158
  13:38:00 STEP  678 H= 169:30 +CPU= 37.913
  13:38:39 STEP  679 H= 169:45 +CPU= 37.759
  13:39:18 STEP  680 H= 170:00 +CPU= 37.923
  13:39:56 STEP  681 H= 170:15 +CPU= 37.343
  13:40:35 STEP  682 H= 170:30 +CPU= 37.811
  13:41:13 STEP  683 H= 170:45 +CPU= 37.618
  13:42:03 STEP  684 H= 171:00 +CPU= 48.298
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000684
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000684
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000684
  13:42:42 STEP  685 H= 171:15 +CPU= 38.745
  13:43:21 STEP  686 H= 171:30 +CPU= 38.442
  13:44:01 STEP  687 H= 171:45 +CPU= 38.779
  13:44:42 STEP  688 H= 172:00 +CPU= 39.909
  13:45:22 STEP  689 H= 172:15 +CPU= 38.804
  13:46:01 STEP  690 H= 172:30 +CPU= 38.451
  13:46:39 STEP  691 H= 172:45 +CPU= 37.091
  13:47:18 STEP  692 H= 173:00 +CPU= 38.058
  13:47:58 STEP  693 H= 173:15 +CPU= 39.009
  13:48:39 STEP  694 H= 173:30 +CPU= 40.247
  13:49:20 STEP  695 H= 173:45 +CPU= 40.253
  13:50:11 STEP  696 H= 174:00 +CPU= 49.563
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000696
  13:50:51 STEP  697 H= 174:15 +CPU= 39.071
  13:51:31 STEP  698 H= 174:30 +CPU= 38.910
  13:52:10 STEP  699 H= 174:45 +CPU= 38.004
  13:52:49 STEP  700 H= 175:00 +CPU= 37.767
  13:53:29 STEP  701 H= 175:15 +CPU= 39.555
  13:54:08 STEP  702 H= 175:30 +CPU= 38.012
  13:54:47 STEP  703 H= 175:45 +CPU= 38.056
  13:55:26 STEP  704 H= 176:00 +CPU= 38.410
  13:56:05 STEP  705 H= 176:15 +CPU= 37.752
  13:56:45 STEP  706 H= 176:30 +CPU= 38.554
  13:57:22 STEP  707 H= 176:45 +CPU= 36.552
  13:58:12 STEP  708 H= 177:00 +CPU= 49.091
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000708
  13:58:53 STEP  709 H= 177:15 +CPU= 39.490
  13:59:33 STEP  710 H= 177:30 +CPU= 39.661
  14:00:15 STEP  711 H= 177:45 +CPU= 40.543
  14:00:54 STEP  712 H= 178:00 +CPU= 38.631
  14:01:34 STEP  713 H= 178:15 +CPU= 39.058
  14:02:13 STEP  714 H= 178:30 +CPU= 37.782
  14:02:51 STEP  715 H= 178:45 +CPU= 37.577
  14:03:31 STEP  716 H= 179:00 +CPU= 39.195
  14:04:09 STEP  717 H= 179:15 +CPU= 36.923
  14:04:50 STEP  718 H= 179:30 +CPU= 40.254
  14:05:32 STEP  719 H= 179:45 +CPU= 40.672
  14:06:26 STEP  720 H= 180:00 +CPU= 53.415
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000720
  14:07:06 STEP  721 H= 180:15 +CPU= 38.327
  14:07:45 STEP  722 H= 180:30 +CPU= 38.405
  14:08:25 STEP  723 H= 180:45 +CPU= 39.049
  14:09:06 STEP  724 H= 181:00 +CPU= 39.720
  14:09:46 STEP  725 H= 181:15 +CPU= 39.282
  14:10:25 STEP  726 H= 181:30 +CPU= 38.416
  14:11:05 STEP  727 H= 181:45 +CPU= 39.063
  14:11:47 STEP  728 H= 182:00 +CPU= 41.056
  14:12:26 STEP  729 H= 182:15 +CPU= 38.445
  14:13:05 STEP  730 H= 182:30 +CPU= 38.230
  14:13:45 STEP  731 H= 182:45 +CPU= 39.108
  14:14:36 STEP  732 H= 183:00 +CPU= 49.595
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000732
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000732
  14:15:15 STEP  733 H= 183:15 +CPU= 37.525
  14:15:55 STEP  734 H= 183:30 +CPU= 38.980
  14:16:35 STEP  735 H= 183:45 +CPU= 39.699
  14:17:16 STEP  736 H= 184:00 +CPU= 40.126
  14:17:56 STEP  737 H= 184:15 +CPU= 38.731
  14:18:34 STEP  738 H= 184:30 +CPU= 37.411
  14:19:14 STEP  739 H= 184:45 +CPU= 38.596
  14:19:55 STEP  740 H= 185:00 +CPU= 40.643
  14:20:38 STEP  741 H= 185:15 +CPU= 41.220
  14:21:19 STEP  742 H= 185:30 +CPU= 40.023
  14:21:59 STEP  743 H= 185:45 +CPU= 39.414
  14:22:48 STEP  744 H= 186:00 +CPU= 47.733
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000744
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000744
  14:23:26 STEP  745 H= 186:15 +CPU= 37.843
  14:24:05 STEP  746 H= 186:30 +CPU= 38.196
  14:24:46 STEP  747 H= 186:45 +CPU= 39.454
  14:25:27 STEP  748 H= 187:00 +CPU= 40.139
  14:26:04 STEP  749 H= 187:15 +CPU= 36.567
  14:26:44 STEP  750 H= 187:30 +CPU= 38.504
  14:27:23 STEP  751 H= 187:45 +CPU= 38.874
  14:28:03 STEP  752 H= 188:00 +CPU= 38.426
  14:28:42 STEP  753 H= 188:15 +CPU= 38.934
  14:29:20 STEP  754 H= 188:30 +CPU= 37.060
  14:30:00 STEP  755 H= 188:45 +CPU= 38.399
  14:30:50 STEP  756 H= 189:00 +CPU= 48.941
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000756
  14:31:29 STEP  757 H= 189:15 +CPU= 38.645
  14:32:09 STEP  758 H= 189:30 +CPU= 39.343
  14:32:50 STEP  759 H= 189:45 +CPU= 39.745
  14:33:31 STEP  760 H= 190:00 +CPU= 39.781
  14:34:09 STEP  761 H= 190:15 +CPU= 37.386
  14:34:46 STEP  762 H= 190:30 +CPU= 36.303
  14:35:24 STEP  763 H= 190:45 +CPU= 37.384
  14:36:04 STEP  764 H= 191:00 +CPU= 38.538
  14:36:42 STEP  765 H= 191:15 +CPU= 37.671
  14:37:23 STEP  766 H= 191:30 +CPU= 39.449
  14:38:02 STEP  767 H= 191:45 +CPU= 38.935
  14:38:57 STEP  768 H= 192:00 +CPU= 53.597
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4s5_2016092300_20_1018_12294524_0_r859296012_7.zip
Uploading the intermediate file: upload_file_7.zip
  14:39:36 STEP  769 H= 192:15 +CPU= 37.296
  14:40:16 STEP  770 H= 192:30 +CPU= 39.197
  14:40:56 STEP  771 H= 192:45 +CPU= 38.911
  14:41:38 STEP  772 H= 193:00 +CPU= 41.574
  14:42:18 STEP  773 H= 193:15 +CPU= 39.264
  14:42:58 STEP  774 H= 193:30 +CPU= 38.784
  14:43:38 STEP  775 H= 193:45 +CPU= 39.302
  14:44:18 STEP  776 H= 194:00 +CPU= 38.667
  14:44:58 STEP  777 H= 194:15 +CPU= 39.742
  14:45:37 STEP  778 H= 194:30 +CPU= 38.054
  14:46:15 STEP  779 H= 194:45 +CPU= 36.988
  14:47:06 STEP  780 H= 195:00 +CPU= 49.341
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000780
  14:47:46 STEP  781 H= 195:15 +CPU= 39.295
  14:48:26 STEP  782 H= 195:30 +CPU= 38.904
  14:49:05 STEP  783 H= 195:45 +CPU= 38.621
  14:49:45 STEP  784 H= 196:00 +CPU= 39.082
  14:50:23 STEP  785 H= 196:15 +CPU= 36.909
  14:51:02 STEP  786 H= 196:30 +CPU= 38.314
  14:51:41 STEP  787 H= 196:45 +CPU= 38.099
  14:52:22 STEP  788 H= 197:00 +CPU= 39.826
  14:53:02 STEP  789 H= 197:15 +CPU= 39.162
  14:53:42 STEP  790 H= 197:30 +CPU= 38.649
  14:54:21 STEP  791 H= 197:45 +CPU= 38.575
  14:55:11 STEP  792 H= 198:00 +CPU= 48.905
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000792
  14:55:52 STEP  793 H= 198:15 +CPU= 39.720
  14:56:32 STEP  794 H= 198:30 +CPU= 39.445
  14:57:11 STEP  795 H= 198:45 +CPU= 38.375
  14:57:52 STEP  796 H= 199:00 +CPU= 39.907
  14:58:32 STEP  797 H= 199:15 +CPU= 39.347
  14:59:13 STEP  798 H= 199:30 +CPU= 39.733
  14:59:53 STEP  799 H= 199:45 +CPU= 38.655
  15:00:34 STEP  800 H= 200:00 +CPU= 40.049
  15:01:14 STEP  801 H= 200:15 +CPU= 39.147
  15:01:52 STEP  802 H= 200:30 +CPU= 37.547
  15:02:33 STEP  803 H= 200:45 +CPU= 40.092
  15:03:25 STEP  804 H= 201:00 +CPU= 50.574
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000804
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000804
  15:04:04 STEP  805 H= 201:15 +CPU= 38.127
  15:04:45 STEP  806 H= 201:30 +CPU= 40.661
  15:05:26 STEP  807 H= 201:45 +CPU= 39.801
  15:06:05 STEP  808 H= 202:00 +CPU= 38.496
  15:06:45 STEP  809 H= 202:15 +CPU= 38.239
  15:07:25 STEP  810 H= 202:30 +CPU= 39.818
  15:08:06 STEP  811 H= 202:45 +CPU= 39.838
  15:08:46 STEP  812 H= 203:00 +CPU= 39.006
  15:09:26 STEP  813 H= 203:15 +CPU= 38.618
  15:10:07 STEP  814 H= 203:30 +CPU= 40.491
  15:10:46 STEP  815 H= 203:45 +CPU= 38.301
  15:11:40 STEP  816 H= 204:00 +CPU= 52.432
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000816
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000816
  15:12:19 STEP  817 H= 204:15 +CPU= 37.995
  15:12:59 STEP  818 H= 204:30 +CPU= 39.019
  15:13:38 STEP  819 H= 204:45 +CPU= 38.060
  15:14:16 STEP  820 H= 205:00 +CPU= 37.636
  15:14:56 STEP  821 H= 205:15 +CPU= 38.432
  15:15:36 STEP  822 H= 205:30 +CPU= 39.218
  15:16:17 STEP  823 H= 205:45 +CPU= 40.037
  15:16:57 STEP  824 H= 206:00 +CPU= 39.121
  15:17:39 STEP  825 H= 206:15 +CPU= 40.946
  15:18:19 STEP  826 H= 206:30 +CPU= 39.175
  15:18:57 STEP  827 H= 206:45 +CPU= 37.412
  15:19:46 STEP  828 H= 207:00 +CPU= 47.888
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000828
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000828
  15:20:26 STEP  829 H= 207:15 +CPU= 38.958
  15:21:05 STEP  830 H= 207:30 +CPU= 37.855
  15:21:44 STEP  831 H= 207:45 +CPU= 37.907
  15:22:23 STEP  832 H= 208:00 +CPU= 38.296
  15:23:01 STEP  833 H= 208:15 +CPU= 37.781
  15:23:40 STEP  834 H= 208:30 +CPU= 38.260
  15:24:20 STEP  835 H= 208:45 +CPU= 38.778
  15:24:59 STEP  836 H= 209:00 +CPU= 38.201
  15:25:39 STEP  837 H= 209:15 +CPU= 39.444
  15:26:20 STEP  838 H= 209:30 +CPU= 40.102
  15:26:58 STEP  839 H= 209:45 +CPU= 37.682
  15:27:47 STEP  840 H= 210:00 +CPU= 48.137
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000840
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000840
  15:28:28 STEP  841 H= 210:15 +CPU= 39.250
  15:29:07 STEP  842 H= 210:30 +CPU= 38.132
  15:29:47 STEP  843 H= 210:45 +CPU= 39.819
  15:30:27 STEP  844 H= 211:00 +CPU= 39.181
  15:31:06 STEP  845 H= 211:15 +CPU= 37.995
  15:31:45 STEP  846 H= 211:30 +CPU= 37.628
  15:32:24 STEP  847 H= 211:45 +CPU= 37.939
  15:33:03 STEP  848 H= 212:00 +CPU= 38.596
  15:33:43 STEP  849 H= 212:15 +CPU= 38.633
  15:34:24 STEP  850 H= 212:30 +CPU= 40.451
  15:35:05 STEP  851 H= 212:45 +CPU= 39.821
  15:35:55 STEP  852 H= 213:00 +CPU= 49.595
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000852
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000852
  15:36:34 STEP  853 H= 213:15 +CPU= 37.519
  15:37:14 STEP  854 H= 213:30 +CPU= 39.435
  15:37:55 STEP  855 H= 213:45 +CPU= 40.231
  15:38:35 STEP  856 H= 214:00 +CPU= 39.049
  15:39:15 STEP  857 H= 214:15 +CPU= 38.675
  15:39:53 STEP  858 H= 214:30 +CPU= 37.145
  15:40:32 STEP  859 H= 214:45 +CPU= 37.942
  15:41:11 STEP  860 H= 215:00 +CPU= 38.553
  15:41:50 STEP  861 H= 215:15 +CPU= 37.340
  15:42:29 STEP  862 H= 215:30 +CPU= 38.816
  15:43:09 STEP  863 H= 215:45 +CPU= 38.484
  15:44:03 STEP  864 H= 216:00 +CPU= 52.385
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000864
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000804
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000816
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000828
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000840
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000852
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000852
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4s5_2016092300_20_1018_12294524_0_r859296012_8.zip
Uploading the intermediate file: upload_file_8.zip
  15:44:43 STEP  865 H= 216:15 +CPU= 39.741
  15:45:24 STEP  866 H= 216:30 +CPU= 39.691
  15:46:03 STEP  867 H= 216:45 +CPU= 37.800
  15:46:44 STEP  868 H= 217:00 +CPU= 40.439
  15:47:23 STEP  869 H= 217:15 +CPU= 37.565
  15:48:03 STEP  870 H= 217:30 +CPU= 39.576
  15:48:45 STEP  871 H= 217:45 +CPU= 40.907
  15:49:26 STEP  872 H= 218:00 +CPU= 40.462
  15:50:06 STEP  873 H= 218:15 +CPU= 38.883
  15:50:44 STEP  874 H= 218:30 +CPU= 37.380
  15:51:23 STEP  875 H= 218:45 +CPU= 38.055
  15:52:15 STEP  876 H= 219:00 +CPU= 50.393
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000876
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000876
  15:52:56 STEP  877 H= 219:15 +CPU= 40.276
  15:53:34 STEP  878 H= 219:30 +CPU= 37.805
  15:54:13 STEP  879 H= 219:45 +CPU= 38.085
  15:54:54 STEP  880 H= 220:00 +CPU= 39.519
  15:55:35 STEP  881 H= 220:15 +CPU= 39.761
  15:56:16 STEP  882 H= 220:30 +CPU= 40.098
  15:56:55 STEP  883 H= 220:45 +CPU= 38.199
  15:57:35 STEP  884 H= 221:00 +CPU= 38.909
  15:58:15 STEP  885 H= 221:15 +CPU= 39.353
  15:58:55 STEP  886 H= 221:30 +CPU= 39.111
  15:59:34 STEP  887 H= 221:45 +CPU= 38.283
  16:00:24 STEP  888 H= 222:00 +CPU= 49.109
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000888
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000888
  16:01:02 STEP  889 H= 222:15 +CPU= 36.910
  16:01:43 STEP  890 H= 222:30 +CPU= 39.857
  16:02:22 STEP  891 H= 222:45 +CPU= 38.445
  16:03:02 STEP  892 H= 223:00 +CPU= 39.042
  16:03:42 STEP  893 H= 223:15 +CPU= 39.142
  16:04:22 STEP  894 H= 223:30 +CPU= 39.166
  16:05:02 STEP  895 H= 223:45 +CPU= 38.807
  16:05:43 STEP  896 H= 224:00 +CPU= 40.550
  16:06:25 STEP  897 H= 224:15 +CPU= 40.886
  16:07:05 STEP  898 H= 224:30 +CPU= 39.217
  16:07:45 STEP  899 H= 224:45 +CPU= 39.026
  16:08:34 STEP  900 H= 225:00 +CPU= 47.223
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000900
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000900
  16:09:15 STEP  901 H= 225:15 +CPU= 40.310
  16:09:56 STEP  902 H= 225:30 +CPU= 40.602
  16:10:35 STEP  903 H= 225:45 +CPU= 37.889
  16:11:13 STEP  904 H= 226:00 +CPU= 37.418
  16:11:54 STEP  905 H= 226:15 +CPU= 39.287
  16:12:33 STEP  906 H= 226:30 +CPU= 38.511
  16:13:12 STEP  907 H= 226:45 +CPU= 38.217
  16:13:52 STEP  908 H= 227:00 +CPU= 38.983
  16:14:30 STEP  909 H= 227:15 +CPU= 36.733
  16:15:08 STEP  910 H= 227:30 +CPU= 37.584
  16:15:48 STEP  911 H= 227:45 +CPU= 38.476
  16:16:40 STEP  912 H= 228:00 +CPU= 51.104
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000912
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000912
  16:17:21 STEP  913 H= 228:15 +CPU= 39.894
  16:18:01 STEP  914 H= 228:30 +CPU= 39.062
  16:18:40 STEP  915 H= 228:45 +CPU= 38.534
  16:19:21 STEP  916 H= 229:00 +CPU= 39.589
  16:20:00 STEP  917 H= 229:15 +CPU= 37.867
  16:20:38 STEP  918 H= 229:30 +CPU= 37.329
  16:21:18 STEP  919 H= 229:45 +CPU= 39.534
  16:21:58 STEP  920 H= 230:00 +CPU= 39.288
  16:22:37 STEP  921 H= 230:15 +CPU= 37.620
  16:23:17 STEP  922 H= 230:30 +CPU= 39.548
  16:23:59 STEP  923 H= 230:45 +CPU= 40.724
  16:24:47 STEP  924 H= 231:00 +CPU= 46.619
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000924
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000924
  16:25:25 STEP  925 H= 231:15 +CPU= 37.600
  16:26:06 STEP  926 H= 231:30 +CPU= 39.937
  16:26:47 STEP  927 H= 231:45 +CPU= 40.499
  16:27:29 STEP  928 H= 232:00 +CPU= 41.337
  16:28:10 STEP  929 H= 232:15 +CPU= 39.812
  16:28:52 STEP  930 H= 232:30 +CPU= 40.711
  16:29:33 STEP  931 H= 232:45 +CPU= 40.650
  16:30:15 STEP  932 H= 233:00 +CPU= 40.553
  16:30:54 STEP  933 H= 233:15 +CPU= 38.406
  16:31:33 STEP  934 H= 233:30 +CPU= 38.173
  16:32:14 STEP  935 H= 233:45 +CPU= 39.697
  16:33:04 STEP  936 H= 234:00 +CPU= 48.546
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000936
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000936
  16:33:44 STEP  937 H= 234:15 +CPU= 39.723
  16:34:25 STEP  938 H= 234:30 +CPU= 39.638
  16:35:04 STEP  939 H= 234:45 +CPU= 38.722
  16:35:45 STEP  940 H= 235:00 +CPU= 40.016
  16:36:25 STEP  941 H= 235:15 +CPU= 38.836
  16:37:05 STEP  942 H= 235:30 +CPU= 39.476
  16:37:46 STEP  943 H= 235:45 +CPU= 39.849
  16:38:27 STEP  944 H= 236:00 +CPU= 40.363
  16:39:07 STEP  945 H= 236:15 +CPU= 38.924
  16:39:46 STEP  946 H= 236:30 +CPU= 37.680
  16:40:26 STEP  947 H= 236:45 +CPU= 38.986
  16:41:17 STEP  948 H= 237:00 +CPU= 49.991
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000948
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000948
  16:41:58 STEP  949 H= 237:15 +CPU= 39.998
  16:42:37 STEP  950 H= 237:30 +CPU= 39.016
  16:43:18 STEP  951 H= 237:45 +CPU= 39.242
  16:43:58 STEP  952 H= 238:00 +CPU= 39.262
  16:44:36 STEP  953 H= 238:15 +CPU= 37.647
  16:45:17 STEP  954 H= 238:30 +CPU= 39.665
  16:45:56 STEP  955 H= 238:45 +CPU= 37.882
  16:46:38 STEP  956 H= 239:00 +CPU= 40.909
  16:47:18 STEP  957 H= 239:15 +CPU= 38.803
  16:47:59 STEP  958 H= 239:30 +CPU= 38.816
  16:48:38 STEP  959 H= 239:45 +CPU= 37.927
  16:49:33 STEP  960 H= 240:00 +CPU= 52.827
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000960
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000864
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000876
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000888
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000900
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000912
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000924
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000936
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMGGh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMSHh7zg+000948
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12294524/ICMUAh7zg+000948
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a4s5_2016092300_20_1018_12294524_0_r859296012_9.zip
Uploading the intermediate file: upload_file_9.zip
Uploading trickle at timestep: 864000
  16:50:15 STEP  961 H= 240:15 +CPU= 39.983
  16:50:56 STEP  962 H= 240:30 +CPU= 39.652
  16:51:37 STEP  963 H= 240:45 +CPU= 39.859
  16:52:18 STEP  964 H= 241:00 +CPU= 40.000
  16:52:58 STEP  965 H= 241:15 +CPU= 38.140
  16:53:37 STEP  966 H= 241:30 +CPU= 37.938
  16:54:18 STEP  967 H= 241:45 +CPU= 38.727
  16:54:59 STEP  968 H= 242:00 +CPU= 40.238
  16:55:41 STEP  969 H= 242:15 +CPU= 40.035
  16:56:22 STEP  970 H= 242:30 +CPU= 39.675
  16:57:02 STEP  971 H= 242:45 +CPU= 39.120
  16:57:53 STEP  972 H= 243:00 +CPU= 49.349
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000972
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000972
  16:58:35 STEP  973 H= 243:15 +CPU= 39.744
  16:59:15 STEP  974 H= 243:30 +CPU= 39.533
  16:59:57 STEP  975 H= 243:45 +CPU= 39.824
  17:00:38 STEP  976 H= 244:00 +CPU= 40.110
  17:01:19 STEP  977 H= 244:15 +CPU= 39.427
  17:02:00 STEP  978 H= 244:30 +CPU= 39.607
  17:02:41 STEP  979 H= 244:45 +CPU= 39.802
  17:03:23 STEP  980 H= 245:00 +CPU= 40.124
  17:04:04 STEP  981 H= 245:15 +CPU= 39.769
  17:04:46 STEP  982 H= 245:30 +CPU= 39.851
  17:05:27 STEP  983 H= 245:45 +CPU= 39.519
  17:06:17 STEP  984 H= 246:00 +CPU= 47.746
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000984
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000984
  17:06:57 STEP  985 H= 246:15 +CPU= 38.589
  17:07:37 STEP  986 H= 246:30 +CPU= 38.465
  17:08:18 STEP  987 H= 246:45 +CPU= 39.561
  17:08:59 STEP  988 H= 247:00 +CPU= 39.689
  17:09:41 STEP  989 H= 247:15 +CPU= 40.451
  17:10:22 STEP  990 H= 247:30 +CPU= 39.727
  17:11:04 STEP  991 H= 247:45 +CPU= 40.297
  17:11:46 STEP  992 H= 248:00 +CPU= 41.185
  17:12:27 STEP  993 H= 248:15 +CPU= 39.111
  17:13:07 STEP  994 H= 248:30 +CPU= 38.783
  17:13:48 STEP  995 H= 248:45 +CPU= 39.650
  17:14:38 STEP  996 H= 249:00 +CPU= 47.336
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+000996
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+000996
  17:15:18 STEP  997 H= 249:15 +CPU= 39.240
  17:16:00 STEP  998 H= 249:30 +CPU= 40.432
  17:16:43 STEP  999 H= 249:45 +CPU= 40.987
  17:17:25 STEP 1000 H= 250:00 +CPU= 41.265
  17:18:08 STEP 1001 H= 250:15 +CPU= 40.906
  17:18:49 STEP 1002 H= 250:30 +CPU= 39.607
  17:19:31 STEP 1003 H= 250:45 +CPU= 40.393
  17:20:13 STEP 1004 H= 251:00 +CPU= 41.180
  17:20:54 STEP 1005 H= 251:15 +CPU= 39.661
  17:21:35 STEP 1006 H= 251:30 +CPU= 39.811
  17:22:16 STEP 1007 H= 251:45 +CPU= 39.064
  17:23:12 STEP 1008 H= 252:00 +CPU= 53.725
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001008
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001008
  17:23:53 STEP 1009 H= 252:15 +CPU= 39.748
  17:24:34 STEP 1010 H= 252:30 +CPU= 39.300
  17:25:15 STEP 1011 H= 252:45 +CPU= 39.874
  17:25:56 STEP 1012 H= 253:00 +CPU= 39.534
  17:26:38 STEP 1013 H= 253:15 +CPU= 40.746
  17:27:19 STEP 1014 H= 253:30 +CPU= 39.799
  17:28:00 STEP 1015 H= 253:45 +CPU= 39.645
  17:28:42 STEP 1016 H= 254:00 +CPU= 40.210
  17:29:25 STEP 1017 H= 254:15 +CPU= 41.503
  17:30:07 STEP 1018 H= 254:30 +CPU= 40.635
  17:30:49 STEP 1019 H= 254:45 +CPU= 40.799
  MAX U WIND=   251.627970120742     
  17:31:41 STEP 1020 H= 255:00 +CPU= 49.672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHh7zg+001020
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAh7zg+001020
  MAX V WIND=   250.234780965483     
  MAX U WIND=   253.817968379884     
  MAX U WIND=   261.337695180095     
  MAX U WIND=   266.014963431065     
  MAX V WIND=   256.209782225870     
  MAX V WIND=   254.324610519663     
  MAX V WIND=   251.425368162466     
  17:32:23 STEP 1021 H= 255:15 +CPU= 40.291
  MAX V WIND=   259.374509386010     
  MAX V WIND=   257.252225938745     
  MAX U WIND=   255.256419217092     
  MAX U WIND=   265.458343088026     
  MAX U WIND=   263.340643470448     
  MAX V WIND=   253.422808254505     
  MAX V WIND=   265.071118968903     
  MAX V WIND=   257.825703049765     
  17:33:04 STEP 1022 H= 255:30 +CPU= 39.466
  MAX V WIND=   255.058503984266     
  MAX U WIND=   252.744236629839     
  MAX U WIND=   266.838494319344     
  MAX U WIND=   254.756847026630     
  MAX V WIND=   260.675882107947     
  MAX V WIND=   257.198311542599     
  MAX V WIND=   255.000023419696     
  17:33:44 STEP 1023 H= 255:45 +CPU= 38.449
  MAX U WIND=   266.288485128390     
  MAX U WIND=   258.288837312937     
  MAX V WIND=   252.492887827012     
  MAX V WIND=   261.600095350238     
  MAX V WIND=   258.005296119643     
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#8 (SIGFPE) :: 4412MB (heap), 5025MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#8 was caused by floating-point overflow [memaddr=0x1cc6692], nsigs = 1
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#8, nsigs = 1
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4412 MB (maxheap), 5025 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173410:1719358450:46013.416] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' : sigcontextptr=0x7ffed0cf7910
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 24) with addr2line-cmd
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x1dd5490] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1cc6692] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x121fd8b] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x1207e29] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x11fdcb2] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0xfac003] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xf5fa6b] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf4dbc4] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xbd7131] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x12c5002] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0xbacbae] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x688153] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x4cd1dc] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4b5d85] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b4f52] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x49d638] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x406fbd] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x401e7f] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e12] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x1dcc34a] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcdbe7] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x401cd5] : ??() at ??:0
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.416] [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' : sigcontextptr=0x7ffed0cf7820
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 25) with addr2line-cmd
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/var/lib/boinc-client/slots/1/oifs_43r3_model.exe' 0x83a8f3 0x83aef9 0x822540 0x1dd5490 0x1cc6692 0x121fd8b 0x1207e29 0x11fdcb2 0xfac003 0xf5fa6b 0xf4dbc4 0xbd7131 0x12c5002 0xbacbae 0x688153 0x4cd1dc 0x4b5d85 0x4b4f52 0x49d638 0x406fbd 0x401e7f 0x401e12 0x1dcc34a 0x1dcdbe7 0x401cd5
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x83a8f3] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x83aef9] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x822540] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x1dd5490] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x1cc6692] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x121fd8b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x1207e29] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x11fdcb2] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0xfac003] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0xf5fa6b] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0xf4dbc4] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0xbd7131] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x12c5002] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0xbacbae] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x688153] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x4cd1dc] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4b5d85] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x4b4f52] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x49d638] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [19]: [0x406fbd] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [20]: [0x401e7f] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [21]: [0x401e12] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [22]: [0x1dcc34a] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [23]: [0x1dcdbe7] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:319] [24]: [0x401cd5] : ??() at ??:0
  [LinuxTraceBack@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.442] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#8, nsigs = 1
[EC_DRHOOK:P5810:1:1:3204511:3204511] [20240625:173411:1719358451:46014.442] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#8, nsigs = 1
forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CC6692  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000121FD8B  vdfmain_                  874  vdfmain.F90
oifs_43r3_model.e  0000000001207E29  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  00000000011FDCB2  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000FAC003  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000F5FA6B  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
..The child process has been killed with signal: 6
>>> Printing last 70 lines from file: NODE.001_01

         AVE   0.412529099550219E-05 0.000000000000000E+00 0.253429906708553E-01
 GPNORM  ICE WATER           AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.658327477037710E-04 0.000000000000000E+00 0.203873400771398E-01
 GPNORM SNOW                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.134540669305771E-03 0.000000000000000E+00 0.293676534765345E-01
 GPNORM RAIN                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.700097352670409E-05 0.000000000000000E+00 0.108246398911600E-01
 GPNORM  CLOUD FRACTION      AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.241331228191935E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM CLW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.409543799711362E-05 0.000000000000000E+00 0.230914383024509E-01
 GPNORM CIW                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.651683925855080E-04 0.000000000000000E+00 0.209312186500887E-01
 GPNORM CC                   AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.240760661092131E+00 0.000000000000000E+00 0.100000000000000E+01
 GPNORM RFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.228125500397997E-04 0.000000000000000E+00 0.285740459299134E-01
 GPNORM SFL                  AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.238244348470151E-04 0.000000000000000E+00 0.674933802144661E-02
 GPNORM CRFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.578221784912677E-05 0.000000000000000E+00 0.164201890218231E-01
 GPNORM CSFL                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.240653977578666E-05 0.000000000000000E+00 0.116713580579802E-01
 GPNORM PFRC                 AVERAGE               MINIMUM               MAXIMUM        
         AVE   0.245483217146191E+00 0.000000000000000E+00 0.100000000000000E+01
 NSTEP =  1020 STEPO   A00000000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1020 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001020 MODE=w
 IO-STREAM CLOSED - ICMGGh7zg+001020
 MPI-TASK:   1 -    316949428 BYTES IN      6 RECORDS TRANSFERRED IN    0.0005 SECONDS633898.9 Mbytes/s, TOTAL TIME=    0.0150( 3.3%)
 NSTEP =  1020 STEPO   0AA000000
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMSHh7zg+001020 MODE=w
 IO-STREAM CLOSED - ./ICMSHh7zg+001020
 MPI-TASK:   1 -    707399250 BYTES IN    202 RECORDS TRANSFERRED IN    0.0123 SECONDS 57512.1 Mbytes/s, TOTAL TIME=    0.4770( 2.6%)
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001020 MODE=w
 IO-STREAM CLOSED - ./ICMUAh7zg+001020
 MPI-TASK:   1 -   5071190848 BYTES IN    125 RECORDS TRANSFERRED IN    0.0078 SECONDS650152.7 Mbytes/s, TOTAL TIME=    0.3598( 2.2%)
 STEPO_FPOS WAS : B00000F00E
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001020 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001020
 MPI-TASK:   1 -           10 BYTES IN      2 RECORDS TRANSFERRED IN    0.0002 SECONDS     0.1 Mbytes/s, TOTAL TIME=    0.0064( 3.1%)
 STEPO_FPOS WAS : V00000000Y
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ./ICMUAh7zg+001020 MODE=a
 IO-STREAM CLOSED - ./ICMUAh7zg+001020
 MPI-TASK:   1 -    308369699 BYTES IN      4 RECORDS TRANSFERRED IN    0.0004 SECONDS770924.3 Mbytes/s, TOTAL TIME=    0.0097( 4.1%)
 STEPO_FPOS WAS : T00000000M
 NSTEP =  1020 STEPO   0AAA00AAA
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1020 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001020 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001020
 MPI-TASK:   1 -    316949428 BYTES IN      7 RECORDS TRANSFERRED IN    0.0006 SECONDS528249.0 Mbytes/s, TOTAL TIME=    0.0165( 3.6%)
 17:31:41 STEP 1020 H= 255:00 +CPU= 49.672
 NSTEP =  1021 STEPO   0AAA00AAA
 17:32:23 STEP 1021 H= 255:15 +CPU= 40.291
 NSTEP =  1022 STEPO   0AAA00AAA
 17:33:04 STEP 1022 H= 255:30 +CPU= 39.466
 NSTEP =  1023 STEPO   0AAA00AAA
 17:33:44 STEP 1023 H= 255:45 +CPU= 38.449
 NSTEP =  1024 STEPO   0AAA00AAA
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat

 17:30:57 FULLPOS-V DYNFPOS   1020     4.811    4.811    4.960    673:09    763:40 0.44033563453421E-04       2GB       0MB
 17:30:58 FULLPOS-T DYNFPOS   1020     0.855    0.855    0.871    673:10    763:41 0.44033563453421E-04       2GB       0MB
 17:30:59 0AAA00AAA STEPO     1020     0.857    0.857    0.879    673:11    763:42 0.44033563453421E-04       2GB       0MB
 17:31:41 0AAA00AAA STEPO     1021    40.638   40.638   42.125    673:51    764:24 0.44033563453421E-04       2GB       0MB
 17:32:23 0AAA00AAA STEPO     1022    40.291   40.291   41.911    674:31    765:06 0.44033563453421E-04       2GB       0MB
 17:33:04 0AAA00AAA STEPO     1023    39.466   39.466   41.000    675:11    765:47 0.44033563453421E-04       2GB       0MB
 17:33:44 0AAA00AAA STEPO     1024    38.449   38.449   39.929    675:49    766:27 0.44033563453421E-04       2GB       0MB
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    1008",
 CTIME   = "00101200      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 2*0,         996, 2*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =  0.000000000000000E+000,
 GMASSI  =  0.000000000000000E+000
 /
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/1/progress_file_12294524.xml

<running_values>
  <last_cpu_time>40579.310000</last_cpu_time>
  <upload_file_number>10</upload_file_number>
  <last_iter>1024</last_iter>
  <last_upload>864000</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
25 Jun 2024 23:28:27 1551031 22448142 oifs_43r3_bl_a4s5_2016092300_20_1018_12294524_0 864,000 37,998 0.0440


©2024 climateprediction.net