Name | oifs_43r3_bl_a0xz_2016092300_15_991_12208010_0 |
Workunit | 12208010 |
Created | 13 Feb 2023, 14:16:02 UTC |
Sent | 13 Feb 2023, 14:18:35 UTC |
Report deadline | 14 Apr 2023, 14:18:35 UTC |
Received | 15 Feb 2023, 12:34:52 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 1 (0x00000001) Unknown error code |
Computer ID | 1537732 |
Run time | 1 hours 57 min 54 sec |
CPU time | 1 hours 57 min 54 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 6.50 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.11 x86_64-pc-linux-gnu |
Peak working set size | 5,484.13 MB |
Peak swap size | 5,930.79 MB |
Peak disk usage | 1,340.94 MB |
Stderr | <core_client_version>7.16.16</core_client_version> <![CDATA[ <message> process exited with code 1 (0x1, -255)</message> <stderr_txt> EP 118 H= 29:30 +CPU= 17.623 18:38:19 STEP 119 H= 29:45 +CPU= 17.542 18:38:41 STEP 120 H= 30:00 +CPU= 21.252 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000120 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000120 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000120 18:39:00 STEP 121 H= 30:15 +CPU= 17.687 18:39:19 STEP 122 H= 30:30 +CPU= 17.610 18:39:37 STEP 123 H= 30:45 +CPU= 17.572 18:39:56 STEP 124 H= 31:00 +CPU= 17.103 18:40:15 STEP 125 H= 31:15 +CPU= 17.414 18:40:34 STEP 126 H= 31:30 +CPU= 17.431 18:40:53 STEP 127 H= 31:45 +CPU= 17.586 18:41:11 STEP 128 H= 32:00 +CPU= 17.019 18:41:30 STEP 129 H= 32:15 +CPU= 17.358 18:41:48 STEP 130 H= 32:30 +CPU= 17.538 18:42:07 STEP 131 H= 32:45 +CPU= 17.511 18:42:30 STEP 132 H= 33:00 +CPU= 21.051 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000132 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000132 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000132 18:42:49 STEP 133 H= 33:15 +CPU= 17.514 18:43:07 STEP 134 H= 33:30 +CPU= 17.501 18:43:26 STEP 135 H= 33:45 +CPU= 17.410 18:43:45 STEP 136 H= 34:00 +CPU= 17.210 18:44:04 STEP 137 H= 34:15 +CPU= 17.544 18:44:22 STEP 138 H= 34:30 +CPU= 17.497 18:44:41 STEP 139 H= 34:45 +CPU= 17.505 18:44:59 STEP 140 H= 35:00 +CPU= 17.083 18:45:18 STEP 141 H= 35:15 +CPU= 17.531 18:45:37 STEP 142 H= 35:30 +CPU= 17.366 18:45:56 STEP 143 H= 35:45 +CPU= 17.401 18:46:24 STEP 144 H= 36:00 +CPU= 22.421 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000144 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000144 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000144 18:46:43 STEP 145 H= 36:15 +CPU= 17.437 18:47:02 STEP 146 H= 36:30 +CPU= 17.496 18:47:20 STEP 147 H= 36:45 +CPU= 17.566 18:47:38 STEP 148 H= 37:00 +CPU= 17.249 18:47:57 STEP 149 H= 37:15 +CPU= 17.787 18:48:16 STEP 150 H= 37:30 +CPU= 17.526 18:48:35 STEP 151 H= 37:45 +CPU= 17.595 18:48:53 STEP 152 H= 38:00 +CPU= 17.383 18:49:12 STEP 153 H= 38:15 +CPU= 17.725 18:49:31 STEP 154 H= 38:30 +CPU= 17.722 18:49:50 STEP 155 H= 38:45 +CPU= 17.392 18:50:13 STEP 156 H= 39:00 +CPU= 21.233 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000156 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000156 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000156 18:50:33 STEP 157 H= 39:15 +CPU= 17.857 18:50:51 STEP 158 H= 39:30 +CPU= 17.841 18:51:09 STEP 159 H= 39:45 +CPU= 17.350 18:51:28 STEP 160 H= 40:00 +CPU= 17.795 18:51:47 STEP 161 H= 40:15 +CPU= 17.858 18:52:05 STEP 162 H= 40:30 +CPU= 17.844 18:52:23 STEP 163 H= 40:45 +CPU= 17.351 18:52:42 STEP 164 H= 41:00 +CPU= 17.779 18:53:01 STEP 165 H= 41:15 +CPU= 17.968 18:53:19 STEP 166 H= 41:30 +CPU= 17.903 18:53:38 STEP 167 H= 41:45 +CPU= 17.587 18:54:00 STEP 168 H= 42:00 +CPU= 21.772 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000168 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000168 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000168 18:54:19 STEP 169 H= 42:15 +CPU= 17.916 18:54:38 STEP 170 H= 42:30 +CPU= 17.863 18:54:55 STEP 171 H= 42:45 +CPU= 17.108 18:55:14 STEP 172 H= 43:00 +CPU= 17.812 18:55:33 STEP 173 H= 43:15 +CPU= 17.857 18:55:51 STEP 174 H= 43:30 +CPU= 17.771 18:56:09 STEP 175 H= 43:45 +CPU= 17.454 18:56:28 STEP 176 H= 44:00 +CPU= 17.677 18:56:46 STEP 177 H= 44:15 +CPU= 17.807 18:57:05 STEP 178 H= 44:30 +CPU= 17.951 18:57:24 STEP 179 H= 44:45 +CPU= 17.564 18:57:46 STEP 180 H= 45:00 +CPU= 21.517 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000180 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000180 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000180 18:58:05 STEP 181 H= 45:15 +CPU= 17.703 18:58:23 STEP 182 H= 45:30 +CPU= 17.799 18:58:42 STEP 183 H= 45:45 +CPU= 17.265 18:59:00 STEP 184 H= 46:00 +CPU= 17.592 18:59:19 STEP 185 H= 46:15 +CPU= 17.103 18:59:40 STEP 186 H= 46:30 +CPU= 15.864 19:00:01 STEP 187 H= 46:45 +CPU= 15.702 19:00:21 STEP 188 H= 47:00 +CPU= 15.745 19:00:42 STEP 189 H= 47:15 +CPU= 15.874 19:01:02 STEP 190 H= 47:30 +CPU= 15.936 19:01:22 STEP 191 H= 47:45 +CPU= 15.500 19:01:53 STEP 192 H= 48:00 +CPU= 21.896 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000192 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000192 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000096 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000096 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000096 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000108 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000108 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000108 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000120 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000120 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000120 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000132 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000132 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000132 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000144 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000144 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000144 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000156 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000156 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000156 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000168 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000180 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000180 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000180 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0xz_2016092300_15_991_12208010_0_r1514058629_1.zip Uploading the intermediate file: upload_file_1.zip 19:02:14 STEP 193 H= 48:15 +CPU= 15.445 19:02:35 STEP 194 H= 48:30 +CPU= 15.907 19:02:53 STEP 195 H= 48:45 +CPU= 15.458 19:03:13 STEP 196 H= 49:00 +CPU= 16.221 19:03:33 STEP 197 H= 49:15 +CPU= 15.936 19:03:53 STEP 198 H= 49:30 +CPU= 15.946 19:04:12 STEP 199 H= 49:45 +CPU= 15.623 19:04:32 STEP 200 H= 50:00 +CPU= 16.000 19:04:51 STEP 201 H= 50:15 +CPU= 16.216 19:05:11 STEP 202 H= 50:30 +CPU= 15.965 19:05:31 STEP 203 H= 50:45 +CPU= 15.667 19:05:55 STEP 204 H= 51:00 +CPU= 19.410 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000204 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000204 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000204 19:06:15 STEP 205 H= 51:15 +CPU= 16.061 19:06:35 STEP 206 H= 51:30 +CPU= 15.945 19:06:55 STEP 207 H= 51:45 +CPU= 15.866 19:07:15 STEP 208 H= 52:00 +CPU= 15.938 19:07:36 STEP 209 H= 52:15 +CPU= 16.115 19:07:56 STEP 210 H= 52:30 +CPU= 16.143 19:08:17 STEP 211 H= 52:45 +CPU= 15.961 19:08:37 STEP 212 H= 53:00 +CPU= 15.939 19:08:58 STEP 213 H= 53:15 +CPU= 16.085 19:09:19 STEP 214 H= 53:30 +CPU= 16.110 19:09:39 STEP 215 H= 53:45 +CPU= 15.855 19:10:05 STEP 216 H= 54:00 +CPU= 19.534 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000216 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000216 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000216 19:10:26 STEP 217 H= 54:15 +CPU= 16.152 19:10:47 STEP 218 H= 54:30 +CPU= 16.078 19:11:07 STEP 219 H= 54:45 +CPU= 15.765 19:11:29 STEP 220 H= 55:00 +CPU= 16.266 19:11:50 STEP 221 H= 55:15 +CPU= 16.176 19:12:11 STEP 222 H= 55:30 +CPU= 16.176 19:12:32 STEP 223 H= 55:45 +CPU= 15.373 19:12:54 STEP 224 H= 56:00 +CPU= 16.258 19:13:15 STEP 225 H= 56:15 +CPU= 15.834 19:13:36 STEP 226 H= 56:30 +CPU= 15.990 19:13:57 STEP 227 H= 56:45 +CPU= 15.436 19:14:24 STEP 228 H= 57:00 +CPU= 20.024 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000228 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000228 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000228 19:14:44 STEP 229 H= 57:15 +CPU= 15.829 19:15:05 STEP 230 H= 57:30 +CPU= 15.646 19:15:25 STEP 231 H= 57:45 +CPU= 15.434 19:15:49 STEP 232 H= 58:00 +CPU= 16.267 19:16:12 STEP 233 H= 58:15 +CPU= 15.899 19:16:34 STEP 234 H= 58:30 +CPU= 15.738 19:16:55 STEP 235 H= 58:45 +CPU= 15.463 19:17:16 STEP 236 H= 59:00 +CPU= 16.009 19:17:36 STEP 237 H= 59:15 +CPU= 15.760 19:17:58 STEP 238 H= 59:30 +CPU= 15.500 19:18:19 STEP 239 H= 59:45 +CPU= 15.661 19:18:52 STEP 240 H= 60:00 +CPU= 21.932 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000240 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000240 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000240 19:19:14 STEP 241 H= 60:15 +CPU= 15.797 19:19:33 STEP 242 H= 60:30 +CPU= 17.637 19:19:51 STEP 243 H= 60:45 +CPU= 18.018 19:20:11 STEP 244 H= 61:00 +CPU= 18.372 19:20:29 STEP 245 H= 61:15 +CPU= 17.702 19:20:48 STEP 246 H= 61:30 +CPU= 17.771 19:21:06 STEP 247 H= 61:45 +CPU= 18.018 19:21:25 STEP 248 H= 62:00 +CPU= 18.282 19:21:44 STEP 249 H= 62:15 +CPU= 17.611 19:22:02 STEP 250 H= 62:30 +CPU= 17.598 19:22:21 STEP 251 H= 62:45 +CPU= 17.945 19:22:44 STEP 252 H= 63:00 +CPU= 22.212 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000252 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000252 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000252 19:23:02 STEP 253 H= 63:15 +CPU= 17.691 19:23:21 STEP 254 H= 63:30 +CPU= 17.654 19:23:40 STEP 255 H= 63:45 +CPU= 18.008 19:23:59 STEP 256 H= 64:00 +CPU= 18.175 19:24:17 STEP 257 H= 64:15 +CPU= 17.642 19:24:35 STEP 258 H= 64:30 +CPU= 17.658 19:24:54 STEP 259 H= 64:45 +CPU= 18.041 19:25:13 STEP 260 H= 65:00 +CPU= 18.119 19:25:31 STEP 261 H= 65:15 +CPU= 17.602 19:25:50 STEP 262 H= 65:30 +CPU= 17.512 19:26:09 STEP 263 H= 65:45 +CPU= 17.820 19:26:32 STEP 264 H= 66:00 +CPU= 22.178 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000264 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000264 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000264 19:26:50 STEP 265 H= 66:15 +CPU= 17.528 19:27:08 STEP 266 H= 66:30 +CPU= 17.619 19:27:27 STEP 267 H= 66:45 +CPU= 17.948 19:27:46 STEP 268 H= 67:00 +CPU= 18.129 19:28:04 STEP 269 H= 67:15 +CPU= 17.574 19:28:23 STEP 270 H= 67:30 +CPU= 17.569 19:28:41 STEP 271 H= 67:45 +CPU= 17.955 19:29:00 STEP 272 H= 68:00 +CPU= 18.150 19:29:19 STEP 273 H= 68:15 +CPU= 17.612 19:29:37 STEP 274 H= 68:30 +CPU= 17.239 19:29:56 STEP 275 H= 68:45 +CPU= 17.933 19:30:20 STEP 276 H= 69:00 +CPU= 22.064 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000276 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000276 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000276 19:30:39 STEP 277 H= 69:15 +CPU= 17.519 19:30:57 STEP 278 H= 69:30 +CPU= 17.505 19:31:16 STEP 279 H= 69:45 +CPU= 17.758 19:31:36 STEP 280 H= 70:00 +CPU= 18.125 19:31:54 STEP 281 H= 70:15 +CPU= 17.571 19:32:13 STEP 282 H= 70:30 +CPU= 17.254 19:32:32 STEP 283 H= 70:45 +CPU= 17.784 19:32:52 STEP 284 H= 71:00 +CPU= 18.020 19:33:11 STEP 285 H= 71:15 +CPU= 17.682 19:33:29 STEP 286 H= 71:30 +CPU= 16.324 19:33:44 STEP 287 H= 71:45 +CPU= 14.676 19:34:25 STEP 288 H= 72:00 +CPU= 20.805 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000288 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000288 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000288 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000192 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000204 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000204 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000204 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000216 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000228 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000240 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000252 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000252 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000252 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000264 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000264 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000264 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMGGh7zg+000276 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMSHh7zg+000276 Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010/ICMUAh7zg+000276 Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a0xz_2016092300_15_991_12208010_0_r1514058629_2.zip 19:34:47 STEP 289 H= 72:15 +CPU= 15.207 Uploading the intermediate file: upload_file_2.zip 19:35:08 STEP 290 H= 72:30 +CPU= 15.206 19:35:29 STEP 291 H= 72:45 +CPU= 15.128 19:35:51 STEP 292 H= 73:00 +CPU= 17.082 19:36:15 STEP 293 H= 73:15 +CPU= 16.588 19:36:36 STEP 294 H= 73:30 +CPU= 16.639 19:36:57 STEP 295 H= 73:45 +CPU= 16.216 19:37:19 STEP 296 H= 74:00 +CPU= 16.818 19:37:40 STEP 297 H= 74:15 +CPU= 16.641 19:38:01 STEP 298 H= 74:30 +CPU= 16.250 19:38:22 STEP 299 H= 74:45 +CPU= 16.394 19:38:50 STEP 300 H= 75:00 +CPU= 20.700 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000300 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000300 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000300 19:39:11 STEP 301 H= 75:15 +CPU= 16.416 19:39:31 STEP 302 H= 75:30 +CPU= 16.439 Quit request received from BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a0xz (argv4) batchid: 991 (argv5) wuid: 12208010 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_33b54a693a29b5642470a7969e91e35a to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0xz_2016092300_15_991_12208010.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0xz_2016092300_15_991_12208010.zip ic_ancil_file: ic_ancil_12208010 ifsdata_file: ifsdata_12208010 climate_data_file: clim_data_12208010 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/2/ic_ancil_12208010.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12208010.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fbd6906a20d3bf25ea2628b2c6c607b9 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12208010.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12208010.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12208010.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12208010.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12208010.xml -- Model is restarting -- Adjusting last_iter, 302, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml last_cpu_time: 6687 upload_file_number: 3 last_iter: 289 last_upload: 259200 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a0xz_2016092300_15_991_12208010_0_r1514058629 The child process has been launched with process id: 410388 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:410388:410388] [20230213:195241:1676314361:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:53:44 STEP 288 H= 72:00 +CPU= 18.407 19:54:04 STEP 289 H= 72:15 +CPU= 16.655 19:54:18 STEP 290 H= 72:30 +CPU= 13.801 19:54:33 STEP 291 H= 72:45 +CPU= 14.240 19:54:53 STEP 292 H= 73:00 +CPU= 17.057 19:55:12 STEP 293 H= 73:15 +CPU= 16.807 19:55:32 STEP 294 H= 73:30 +CPU= 16.705 19:55:55 STEP 295 H= 73:45 +CPU= 16.880 19:56:16 STEP 296 H= 74:00 +CPU= 17.903 19:56:37 STEP 297 H= 74:15 +CPU= 18.014 19:56:59 STEP 298 H= 74:30 +CPU= 17.528 19:57:19 STEP 299 H= 74:45 +CPU= 16.166 19:57:50 STEP 300 H= 75:00 +CPU= 21.202 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGh7zg+000300 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHh7zg+000300 Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAh7zg+000300 19:58:10 STEP 301 H= 75:15 +CPU= 16.742 19:58:30 STEP 302 H= 75:30 +CPU= 16.380 19:58:52 STEP 303 H= 75:45 +CPU= 16.733 19:59:12 STEP 304 H= 76:00 +CPU= 18.090 (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a0xz (argv4) batchid: 991 (argv5) wuid: 12208010 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_33b54a693a29b5642470a7969e91e35a to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0xz_2016092300_15_991_12208010.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0xz_2016092300_15_991_12208010.zip ic_ancil_file: ic_ancil_12208010 ifsdata_file: ifsdata_12208010 climate_data_file: clim_data_12208010 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/2/ic_ancil_12208010.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12208010.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fbd6906a20d3bf25ea2628b2c6c607b9 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12208010.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12208010.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12208010.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12208010.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12208010.xml -- Model is restarting -- Adjusting last_iter, 305, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml last_cpu_time: 6992 upload_file_number: 3 last_iter: 289 last_upload: 259200 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a0xz_2016092300_15_991_12208010_0_r1514058629 The child process has been launched with process id: 61675 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133146:1676377906:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 13:32:43 STEP 288 H= 72:00 +CPU= 18.265 13:36:42 STEP 289 H= 72:15 +CPU= 14.585 13:36:56 STEP 290 H= 72:30 +CPU= 11.507 13:37:07 STEP 291 H= 72:45 +CPU= 11.590 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133710:1676378230:324.563] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5372MB (heap), 5278MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.851] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.851] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f0c0 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.851] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.851] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.851] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3843] 5372 MB (maxheap), 5278 MB (maxrss), 0 MB (maxstack) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : MASTER [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT0 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT2 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT3 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CNT4 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : STEPO [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : SCAN2M [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : GP_MODEL [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : CALL_SL [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LAPINEA [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARMES [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LARCINA [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133711:1676378231:324.933] [c_drhook_print_@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:3897] : LAITLI double free or corruption (top) [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133712:1676378232:325.933] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1 [EC_DRHOOK:linux-tower:1:1:61675:61675] [20230214:133712:1676378232:325.933] [signal_drhook@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x401140 for signal#15, nsigs = 1 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source oifs_43r3_model.e 000000000081FCF1 Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8BB20 Unknown Unknown Unknown oifs_43r3_model.e 0000000000FDDE0D laitli_ 152 laitli.F90 oifs_43r3_model.e 0000000001065984 larcina_ 405 larcina.F90 oifs_43r3_model.e 0000000001062038 larmes_ 603 larmes.F90 oifs_43r3_model.e 0000000001058F5D lapinea_ 270 lapinea.F90 oifs_43r3_model.e 0000000000FF4E6C call_sl_ 300 call_sl.F90 oifs_43r3_model.e 0000000001081A7D call_sl_heap_ 94 call_sl_heap.F90 oifs_43r3_model.e 0000000000E67648 gp_model_ 533 gp_model.F90 oifs_43r3_model.e 00000000015C1E62 gp_model_heap_ 74 gp_model_heap.F90 oifs_43r3_model.e 00000000015AC40E scan2m_ 535 scan2m.F90 oifs_43r3_model.e 00000000006620B4 stepo_ 327 stepo.F90 oifs_43r3_model.e 00000000004A0E7C cnt4_ 1133 cnt4.F90 oifs_43r3_model.e 000000000049DC29 cnt3_ 267 cnt3.F90 oifs_43r3_model.e 000000000049CE72 cnt2_ 88 cnt2.F90 oifs_43r3_model.e 000000000049CD68 cnt1_ 92 cnt1.F90 oifs_43r3_model.e 00000000004078AD cnt0_ 146 cnt0.F90 oifs_43r3_model.e 000000000040276F MAIN__ 96 master.F90 oifs_43r3_model.e 000000000040270D Unknown Unknown Unknown oifs_43r3_model.e 0000000001D8287A Unknown Unknown Unknown oifs_43r3_model.e 0000000001D840D7 Unknown Unknown Unknown oifs_43r3_model.e 00000000004025E5 Unknown Unknown Unknown (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.11_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a0xz (argv4) batchid: 991 (argv5) wuid: 12208010 (argv6) fclen: 15 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /var/lib/boinc-client/slots/2 Project directory is: /var/lib/boinc-client/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.11 Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12208010 ..mkdir for temp folder for results failed Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_app_1.11_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_33b54a693a29b5642470a7969e91e35a to: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0xz_2016092300_15_991_12208010.zip Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_bl_a0xz_2016092300_15_991_12208010.zip ic_ancil_file: ic_ancil_12208010 ifsdata_file: ifsdata_12208010 climate_data_file: clim_data_12208010 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc-client/slots/2/ic_ancil_12208010.zip Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12208010.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_fbd6906a20d3bf25ea2628b2c6c607b9 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12208010.zip Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12208010.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12208010.zip Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12208010.zip Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12208010.xml -- Model is restarting -- Adjusting last_iter, 292, to previous model restart step. Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml last_cpu_time: 7050 upload_file_number: 3 last_iter: 289 last_upload: 259200 model_completed: 0 total_length_of_simulation: 1296000 result_base_name: oifs_43r3_bl_a0xz_2016092300_15_991_12208010_0_r1514058629 The child process has been launched with process id: 404987 Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2091] fp = 0x30e4a00 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [process_options@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [ignore_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [restore_default_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [trapfpe_treatment@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f520 (old at 0x401140) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [signal_drhook_init@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f520 (old at (nil)) [EC_DRHOOK:linux-tower:1:1:404987:404987] [20230214:193212:1676399532:0.000] [catch_signals@/home/abowery/Desktop/OpenIFS/oifs_43r3_bl/gc_oifs43r3-feature-cpdn/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 19:33:04 STEP 288 H= 72:00 +CPU= 18.371 ..The child process has been killed with signal: 9 >>> Printing last 70 lines from file: NODE.001_01 Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Organic_Matter_hydrophilic(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Organic_Matter_hydrophobic(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Mineral_Dust_bin1(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Mineral_Dust_bin2(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Mineral_Dust_bin3(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sea_Salt_bin1(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sea_Salt_bin2(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sea_Salt_bin3(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sulfates(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc UPDTIM MACC AEROSOL CLIMATOLOGY IS LOADED SUECAEC: INLEV,ISYEAR,IEYEAR= 0 1957 2010 suecaec NAERMACC = 1 UPDT RAESHDU = 2000.0 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20160923 90 9.999999999999998E-005 DATE= 2016 9 26 NORMS AT NSTEP CNT4 288 SPECTRAL NORMS - LOG(PREHYDS) 0.115129254648196E+02 OROGRAPHY 0.000000000000000E+00 LEV VORTICITY DIVERGENCE TEMPERATURE KINETIC ENERGY AVE 0.135534115664460E-05 0.261218884847622E-06 0.207367232180281E+03 0.484701387712131E+01 GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM AVE 0.634332115110589E-03 0.100000000000000E-07 0.774120538734860E-02 GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM AVE 0.982237267573109E-08 0.000000000000000E+00 0.141297970281623E-06 GPNORM SNOW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM RAIN AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM AVE 0.709422669786545E-01 0.000000000000000E+00 0.100000000000000E+01 GPNORM CLW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CIW AVERAGE MINIMUM MAXIMUM AVE 0.982045920090044E-08 0.000000000000000E+00 0.141300081567565E-06 GPNORM CC AVERAGE MINIMUM MAXIMUM AVE 0.708836616366385E-01 0.000000000000000E+00 0.100000000000000E+01 GPNORM RFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM SFL AVERAGE MINIMUM MAXIMUM AVE 0.169559401853914E-10 0.000000000000000E+00 0.298452118735921E-09 GPNORM CRFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CSFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM PFRC AVERAGE MINIMUM MAXIMUM AVE 0.101238092999973E-01 0.000000000000000E+00 0.100000000000000E+01 NSTEP = 288 STEPO 0AAA00AAA 19:33:04 STEP 288 H= 72:00 +CPU= 18.371 NSTEP = 289 STEPO 0AAA00AAA ------------------------------------------------ CNT0 not found; string returned was: 'STEPO' >>> Printing last 3 lines from file: ifs.stat 19:32:12 000000000 CNT3 -999 0.209 0.209 0.210 0:00 0:00 0.00000000000000E+00 1GB 0MB 19:32:40 0AAA00AAA STEPO 288 2.419 2.419 28.263 0:02 0:28 0.26121888484762E-06 1GB 0MB 19:33:04 0AAA00AAA STEPO 289 18.206 18.206 23.394 0:20 0:51 0.26121888484762E-06 2GB 0MB ------------------------------------------------ >>> Printing last 11 lines from file: rcf CSTEP = " 288", CTIME = "00030000 ", NSTEPLPP = 201, 202, 49, 228226, 228227, 2*0, 276, 2*0, IPRGPNSRES = 1, IPRGPEWRES = 1, IPRTRWRES = 1, IPRTRVRES = 1, GMASS0 = 0.000000000000000E+000, GMASSI = 0.000000000000000E+000 / ------------------------------------------------ >>> Printing last 8 lines from file: /var/lib/boinc-client/slots/2/progress_file_12208010.xml <running_values> <last_cpu_time>7074.010000</last_cpu_time> <upload_file_number>3</upload_file_number> <last_iter>289</last_iter> <last_upload>259200</last_upload> <model_completed>0</model_completed> </running_values> ------------------------------------------------ ..Failed, model did not complete successfully </stderr_txt> ]]> |
No trickles! |
---|
©2024 cpdn.org