Name | oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0 |
Workunit | 12290796 |
Created | 12 Jun 2024, 23:36:57 UTC |
Sent | 12 Jun 2024, 23:45:49 UTC |
Report deadline | 11 Aug 2024, 23:45:49 UTC |
Received | 21 Jun 2024, 1:16:41 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 151 (0x00000097) Unknown error code |
Computer ID | 1545393 |
Run time | 1 hours 37 min 18 sec |
CPU time | 1 hours 37 min 18 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 8.06 GFLOPS |
Application version | OpenIFS 43r3 Baroclinic Lifecycle v1.13 x86_64-pc-linux-gnu |
Peak working set size | 5.43 GB |
Peak swap size | 5.84 GB |
Peak disk usage | 1.31 GB |
Stderr | <core_client_version>7.22.1</core_client_version> <![CDATA[ <message> process exited with code 151 (0x97, -105)</message> <stderr_txt> H= 69:15 +CPU= 7.949 03:03:59 STEP 278 H= 69:30 +CPU= 7.914 03:04:07 STEP 279 H= 69:45 +CPU= 7.824 03:04:15 STEP 280 H= 70:00 +CPU= 8.002 03:04:23 STEP 281 H= 70:15 +CPU= 7.874 03:04:31 STEP 282 H= 70:30 +CPU= 7.828 03:04:38 STEP 283 H= 70:45 +CPU= 7.850 03:04:46 STEP 284 H= 71:00 +CPU= 7.975 03:04:54 STEP 285 H= 71:15 +CPU= 7.868 03:05:02 STEP 286 H= 71:30 +CPU= 7.801 03:05:10 STEP 287 H= 71:45 +CPU= 7.894 03:05:21 STEP 288 H= 72:00 +CPU= 11.051 03:05:29 STEP 289 H= 72:15 +CPU= 7.961 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000288 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000288 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000288 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000192 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000192 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000192 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000204 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000204 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000204 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000216 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000216 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000216 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000228 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000228 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000228 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000240 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000240 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000240 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000252 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000252 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000252 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000264 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000264 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000264 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000276 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000276 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000276 Zipping up the intermediate file: /home/seb/projects/climateprediction.net/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0_r1129607049_2.zip Uploading the intermediate file: upload_file_2.zip 03:05:37 STEP 290 H= 72:30 +CPU= 8.032 03:05:45 STEP 291 H= 72:45 +CPU= 7.879 03:05:53 STEP 292 H= 73:00 +CPU= 8.014 03:06:01 STEP 293 H= 73:15 +CPU= 7.846 03:06:09 STEP 294 H= 73:30 +CPU= 7.895 03:06:17 STEP 295 H= 73:45 +CPU= 7.848 03:06:25 STEP 296 H= 74:00 +CPU= 8.005 03:06:33 STEP 297 H= 74:15 +CPU= 7.808 03:06:41 STEP 298 H= 74:30 +CPU= 7.797 03:06:49 STEP 299 H= 74:45 +CPU= 7.838 03:06:59 STEP 300 H= 75:00 +CPU= 10.047 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000300 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000300 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000300 03:07:07 STEP 301 H= 75:15 +CPU= 7.846 03:07:15 STEP 302 H= 75:30 +CPU= 7.876 03:07:23 STEP 303 H= 75:45 +CPU= 7.800 03:07:31 STEP 304 H= 76:00 +CPU= 7.972 03:07:39 STEP 305 H= 76:15 +CPU= 7.869 03:07:46 STEP 306 H= 76:30 +CPU= 7.811 03:07:54 STEP 307 H= 76:45 +CPU= 7.833 03:08:02 STEP 308 H= 77:00 +CPU= 8.017 03:08:10 STEP 309 H= 77:15 +CPU= 7.830 03:08:18 STEP 310 H= 77:30 +CPU= 7.799 03:08:26 STEP 311 H= 77:45 +CPU= 7.884 03:08:36 STEP 312 H= 78:00 +CPU= 9.994 03:08:44 STEP 313 H= 78:15 +CPU= 7.802 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000312 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000312 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000312 03:08:52 STEP 314 H= 78:30 +CPU= 7.852 03:09:00 STEP 315 H= 78:45 +CPU= 7.796 03:09:08 STEP 316 H= 79:00 +CPU= 7.915 03:09:15 STEP 317 H= 79:15 +CPU= 7.761 03:09:23 STEP 318 H= 79:30 +CPU= 7.849 03:09:31 STEP 319 H= 79:45 +CPU= 7.835 03:09:39 STEP 320 H= 80:00 +CPU= 8.172 03:09:47 STEP 321 H= 80:15 +CPU= 7.840 03:09:55 STEP 322 H= 80:30 +CPU= 7.858 03:10:03 STEP 323 H= 80:45 +CPU= 7.873 03:10:13 STEP 324 H= 81:00 +CPU= 10.018 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000324 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000324 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000324 03:10:21 STEP 325 H= 81:15 +CPU= 7.933 03:10:29 STEP 326 H= 81:30 +CPU= 7.979 03:10:37 STEP 327 H= 81:45 +CPU= 7.899 03:10:46 STEP 328 H= 82:00 +CPU= 8.256 03:10:54 STEP 329 H= 82:15 +CPU= 7.927 03:11:02 STEP 330 H= 82:30 +CPU= 7.943 03:11:10 STEP 331 H= 82:45 +CPU= 8.367 03:11:19 STEP 332 H= 83:00 +CPU= 8.532 03:11:27 STEP 333 H= 83:15 +CPU= 8.133 03:11:35 STEP 334 H= 83:30 +CPU= 7.894 03:11:43 STEP 335 H= 83:45 +CPU= 8.027 03:11:54 STEP 336 H= 84:00 +CPU= 11.207 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000336 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000336 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000336 03:12:02 STEP 337 H= 84:15 +CPU= 7.771 03:12:10 STEP 338 H= 84:30 +CPU= 8.213 03:12:18 STEP 339 H= 84:45 +CPU= 7.930 03:12:26 STEP 340 H= 85:00 +CPU= 8.140 03:12:34 STEP 341 H= 85:15 +CPU= 7.875 03:12:42 STEP 342 H= 85:30 +CPU= 7.865 03:12:50 STEP 343 H= 85:45 +CPU= 7.867 03:12:58 STEP 344 H= 86:00 +CPU= 8.011 03:13:06 STEP 345 H= 86:15 +CPU= 7.883 03:13:14 STEP 346 H= 86:30 +CPU= 7.822 03:13:22 STEP 347 H= 86:45 +CPU= 7.771 03:13:32 STEP 348 H= 87:00 +CPU= 10.027 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000348 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000348 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000348 03:13:40 STEP 349 H= 87:15 +CPU= 7.801 03:13:48 STEP 350 H= 87:30 +CPU= 7.729 03:13:55 STEP 351 H= 87:45 +CPU= 7.796 03:14:04 STEP 352 H= 88:00 +CPU= 8.074 03:14:12 STEP 353 H= 88:15 +CPU= 7.925 03:14:20 STEP 354 H= 88:30 +CPU= 7.881 03:14:27 STEP 355 H= 88:45 +CPU= 7.818 03:14:35 STEP 356 H= 89:00 +CPU= 7.939 03:14:43 STEP 357 H= 89:15 +CPU= 7.858 03:14:51 STEP 358 H= 89:30 +CPU= 7.786 03:14:59 STEP 359 H= 89:45 +CPU= 7.811 03:15:09 STEP 360 H= 90:00 +CPU= 9.962 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000360 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000360 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000360 03:15:17 STEP 361 H= 90:15 +CPU= 7.866 03:15:25 STEP 362 H= 90:30 +CPU= 7.896 03:15:33 STEP 363 H= 90:45 +CPU= 7.860 03:15:41 STEP 364 H= 91:00 +CPU= 8.005 03:15:49 STEP 365 H= 91:15 +CPU= 7.804 03:15:57 STEP 366 H= 91:30 +CPU= 7.831 03:16:04 STEP 367 H= 91:45 +CPU= 7.792 03:16:12 STEP 368 H= 92:00 +CPU= 7.954 03:16:20 STEP 369 H= 92:15 +CPU= 7.844 03:16:28 STEP 370 H= 92:30 +CPU= 7.849 03:16:36 STEP 371 H= 92:45 +CPU= 7.826 03:16:46 STEP 372 H= 93:00 +CPU= 9.954 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000372 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000372 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000372 03:16:54 STEP 373 H= 93:15 +CPU= 7.802 03:17:02 STEP 374 H= 93:30 +CPU= 7.879 03:17:10 STEP 375 H= 93:45 +CPU= 7.869 03:17:18 STEP 376 H= 94:00 +CPU= 8.031 03:17:26 STEP 377 H= 94:15 +CPU= 7.885 03:17:34 STEP 378 H= 94:30 +CPU= 7.906 03:17:42 STEP 379 H= 94:45 +CPU= 7.902 03:17:50 STEP 380 H= 95:00 +CPU= 8.029 03:17:58 STEP 381 H= 95:15 +CPU= 7.879 03:18:06 STEP 382 H= 95:30 +CPU= 7.873 03:18:14 STEP 383 H= 95:45 +CPU= 7.987 03:18:25 STEP 384 H= 96:00 +CPU= 10.981 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000384 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000384 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000384 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000288 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000288 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000288 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000300 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000300 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000300 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000312 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000312 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000312 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000324 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000324 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000324 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000336 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000336 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000336 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000348 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000348 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000348 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000360 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000360 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000360 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000372 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000372 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000372 Zipping up the intermediate file: /home/seb/projects/climateprediction.net/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0_r1129607049_3.zip Uploading the intermediate file: upload_file_3.zip 03:18:33 STEP 385 H= 96:15 +CPU= 7.997 03:18:41 STEP 386 H= 96:30 +CPU= 8.022 03:18:49 STEP 387 H= 96:45 +CPU= 7.968 03:18:57 STEP 388 H= 97:00 +CPU= 8.003 03:19:05 STEP 389 H= 97:15 +CPU= 7.833 03:19:13 STEP 390 H= 97:30 +CPU= 7.853 03:19:21 STEP 391 H= 97:45 +CPU= 7.888 03:19:29 STEP 392 H= 98:00 +CPU= 8.069 03:19:37 STEP 393 H= 98:15 +CPU= 7.934 03:19:45 STEP 394 H= 98:30 +CPU= 7.905 03:19:53 STEP 395 H= 98:45 +CPU= 7.993 03:20:03 STEP 396 H= 99:00 +CPU= 9.869 03:20:11 STEP 397 H= 99:15 +CPU= 7.845 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000396 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000396 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000396 03:20:18 STEP 398 H= 99:30 +CPU= 7.876 03:20:26 STEP 399 H= 99:45 +CPU= 7.868 03:20:35 STEP 400 H= 100:00 +CPU= 8.086 03:20:42 STEP 401 H= 100:15 +CPU= 7.669 03:20:50 STEP 402 H= 100:30 +CPU= 7.945 03:20:58 STEP 403 H= 100:45 +CPU= 7.928 03:21:06 STEP 404 H= 101:00 +CPU= 8.140 03:21:14 STEP 405 H= 101:15 +CPU= 7.882 03:21:22 STEP 406 H= 101:30 +CPU= 7.840 03:21:30 STEP 407 H= 101:45 +CPU= 7.820 03:21:40 STEP 408 H= 102:00 +CPU= 9.937 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000408 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000408 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000408 03:21:48 STEP 409 H= 102:15 +CPU= 7.861 03:21:56 STEP 410 H= 102:30 +CPU= 7.918 03:22:04 STEP 411 H= 102:45 +CPU= 7.925 03:22:12 STEP 412 H= 103:00 +CPU= 8.139 03:22:20 STEP 413 H= 103:15 +CPU= 7.948 03:22:28 STEP 414 H= 103:30 +CPU= 7.865 03:22:36 STEP 415 H= 103:45 +CPU= 7.884 03:22:44 STEP 416 H= 104:00 +CPU= 8.069 03:22:52 STEP 417 H= 104:15 +CPU= 7.757 03:23:00 STEP 418 H= 104:30 +CPU= 7.796 03:23:08 STEP 419 H= 104:45 +CPU= 7.827 03:23:18 STEP 420 H= 105:00 +CPU= 10.135 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000420 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000420 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000420 03:23:26 STEP 421 H= 105:15 +CPU= 7.923 03:23:34 STEP 422 H= 105:30 +CPU= 7.879 03:23:42 STEP 423 H= 105:45 +CPU= 7.854 03:23:50 STEP 424 H= 106:00 +CPU= 8.094 03:23:58 STEP 425 H= 106:15 +CPU= 7.877 03:24:05 STEP 426 H= 106:30 +CPU= 7.743 03:24:13 STEP 427 H= 106:45 +CPU= 7.808 03:24:21 STEP 428 H= 107:00 +CPU= 8.072 03:24:29 STEP 429 H= 107:15 +CPU= 7.881 03:24:37 STEP 430 H= 107:30 +CPU= 7.878 03:24:45 STEP 431 H= 107:45 +CPU= 7.899 03:24:56 STEP 432 H= 108:00 +CPU= 11.012 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000432 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000432 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000432 03:25:04 STEP 433 H= 108:15 +CPU= 7.844 03:25:12 STEP 434 H= 108:30 +CPU= 7.797 03:25:20 STEP 435 H= 108:45 +CPU= 7.970 03:25:28 STEP 436 H= 109:00 +CPU= 8.096 03:25:36 STEP 437 H= 109:15 +CPU= 7.892 03:25:44 STEP 438 H= 109:30 +CPU= 7.813 03:25:52 STEP 439 H= 109:45 +CPU= 7.902 03:26:00 STEP 440 H= 110:00 +CPU= 8.039 03:26:08 STEP 441 H= 110:15 +CPU= 7.818 03:26:16 STEP 442 H= 110:30 +CPU= 7.712 03:26:24 STEP 443 H= 110:45 +CPU= 7.871 03:26:34 STEP 444 H= 111:00 +CPU= 10.004 03:26:42 STEP 445 H= 111:15 +CPU= 7.959 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000444 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000444 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000444 03:26:50 STEP 446 H= 111:30 +CPU= 7.910 03:26:58 STEP 447 H= 111:45 +CPU= 7.840 03:27:06 STEP 448 H= 112:00 +CPU= 7.982 03:27:14 STEP 449 H= 112:15 +CPU= 7.855 03:27:21 STEP 450 H= 112:30 +CPU= 7.852 03:27:29 STEP 451 H= 112:45 +CPU= 7.814 03:27:37 STEP 452 H= 113:00 +CPU= 8.042 03:27:45 STEP 453 H= 113:15 +CPU= 7.775 03:27:53 STEP 454 H= 113:30 +CPU= 7.819 03:28:01 STEP 455 H= 113:45 +CPU= 7.763 03:28:11 STEP 456 H= 114:00 +CPU= 10.096 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000456 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000456 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000456 03:28:19 STEP 457 H= 114:15 +CPU= 7.788 03:28:27 STEP 458 H= 114:30 +CPU= 7.817 03:28:35 STEP 459 H= 114:45 +CPU= 7.869 03:28:43 STEP 460 H= 115:00 +CPU= 8.004 03:28:51 STEP 461 H= 115:15 +CPU= 7.839 03:28:58 STEP 462 H= 115:30 +CPU= 7.777 03:29:06 STEP 463 H= 115:45 +CPU= 7.830 03:29:14 STEP 464 H= 116:00 +CPU= 8.005 03:29:22 STEP 465 H= 116:15 +CPU= 7.851 03:29:30 STEP 466 H= 116:30 +CPU= 7.759 03:29:38 STEP 467 H= 116:45 +CPU= 7.832 03:29:48 STEP 468 H= 117:00 +CPU= 9.950 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000468 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000468 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000468 03:29:56 STEP 469 H= 117:15 +CPU= 7.802 03:30:04 STEP 470 H= 117:30 +CPU= 7.782 03:30:11 STEP 471 H= 117:45 +CPU= 7.839 03:30:19 STEP 472 H= 118:00 +CPU= 7.966 03:30:27 STEP 473 H= 118:15 +CPU= 7.836 03:30:35 STEP 474 H= 118:30 +CPU= 7.875 03:30:43 STEP 475 H= 118:45 +CPU= 7.878 03:30:51 STEP 476 H= 119:00 +CPU= 8.107 03:30:59 STEP 477 H= 119:15 +CPU= 7.821 03:31:07 STEP 478 H= 119:30 +CPU= 7.828 03:31:15 STEP 479 H= 119:45 +CPU= 7.895 03:31:26 STEP 480 H= 120:00 +CPU= 11.026 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000480 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000480 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000480 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000384 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000384 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000384 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000396 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000396 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000396 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000408 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000408 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000408 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000420 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000420 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000420 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000432 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000432 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000432 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000444 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000444 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000444 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000456 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000456 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000456 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000468 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000468 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000468 Zipping up the intermediate file: /home/seb/projects/climateprediction.net/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0_r1129607049_4.zip 03:31:34 STEP 481 H= 120:15 +CPU= 7.895 Uploading the intermediate file: upload_file_4.zip 03:31:42 STEP 482 H= 120:30 +CPU= 7.919 03:31:50 STEP 483 H= 120:45 +CPU= 7.869 03:31:58 STEP 484 H= 121:00 +CPU= 8.100 03:32:06 STEP 485 H= 121:15 +CPU= 7.852 03:32:14 STEP 486 H= 121:30 +CPU= 7.783 03:32:22 STEP 487 H= 121:45 +CPU= 7.865 03:32:30 STEP 488 H= 122:00 +CPU= 7.972 03:32:38 STEP 489 H= 122:15 +CPU= 7.777 03:32:46 STEP 490 H= 122:30 +CPU= 7.763 03:32:53 STEP 491 H= 122:45 +CPU= 7.867 03:33:04 STEP 492 H= 123:00 +CPU= 10.032 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000492 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000492 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000492 03:33:11 STEP 493 H= 123:15 +CPU= 7.838 03:33:19 STEP 494 H= 123:30 +CPU= 7.849 03:33:27 STEP 495 H= 123:45 +CPU= 8.006 03:33:35 STEP 496 H= 124:00 +CPU= 8.054 03:33:43 STEP 497 H= 124:15 +CPU= 7.836 03:33:51 STEP 498 H= 124:30 +CPU= 7.756 03:33:59 STEP 499 H= 124:45 +CPU= 7.870 03:34:07 STEP 500 H= 125:00 +CPU= 8.085 03:34:15 STEP 501 H= 125:15 +CPU= 7.912 03:34:23 STEP 502 H= 125:30 +CPU= 7.830 03:34:31 STEP 503 H= 125:45 +CPU= 7.886 03:34:41 STEP 504 H= 126:00 +CPU= 10.035 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000504 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000504 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000504 03:34:49 STEP 505 H= 126:15 +CPU= 7.818 03:34:57 STEP 506 H= 126:30 +CPU= 7.834 03:35:05 STEP 507 H= 126:45 +CPU= 7.814 03:35:13 STEP 508 H= 127:00 +CPU= 8.018 03:35:21 STEP 509 H= 127:15 +CPU= 7.957 03:35:29 STEP 510 H= 127:30 +CPU= 7.730 03:35:36 STEP 511 H= 127:45 +CPU= 7.805 03:35:44 STEP 512 H= 128:00 +CPU= 7.955 03:35:52 STEP 513 H= 128:15 +CPU= 7.711 03:36:00 STEP 514 H= 128:30 +CPU= 7.759 03:36:08 STEP 515 H= 128:45 +CPU= 7.876 03:36:18 STEP 516 H= 129:00 +CPU= 9.897 03:36:26 STEP 517 H= 129:15 +CPU= 7.761 Suspend request received from the BOINC client, suspending the child process Quit request received from the BOINC client, ending the child process (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a1wl (argv4) batchid: 1018 (argv5) wuid: 12290796 (argv6) fclen: 20 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /home/seb/slots/0 Project directory is: /home/seb/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.13 Location of temp folder: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796 ..mkdir for temp folder for results failed Copying: /home/seb/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/seb/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /home/seb/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_6c71865a3882b3b62f5c7f2394aa73be to: /home/seb/slots/0/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796.zip Unzipping the namelist zip file: /home/seb/slots/0/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796.zip ic_ancil_file: ic_ancil_12290796 ifsdata_file: ifsdata_12290796 climate_data_file: clim_data_12290796 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/seb/slots/0/ic_ancil_12290796.zip Unzipping the IC ancils zip file: /home/seb/slots/0/ic_ancil_12290796.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5a377634e1417298b1132a8f0e110fb8 to: /home/seb/slots/0/ifsdata/ifsdata_12290796.zip Unzipping the ifsdata_zip file: /home/seb/slots/0/ifsdata/ifsdata_12290796.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/seb/slots/0/159l_2/clim_data_12290796.zip Unzipping the climate data zip file: /home/seb/slots/0/159l_2/clim_data_12290796.zip Checking for progress XML file: /home/seb/slots/0/progress_file_12290796.xml Opened progress file ok : /home/seb/slots/0/progress_file_12290796.xml -- Model is restarting -- Adjusting last_iter, 518, to previous model restart step. Creating progress file: /home/seb/slots/0/progress_file_12290796.xml last_cpu_time: 4669 upload_file_number: 5 last_iter: 481 last_upload: 432000 model_completed: 0 total_length_of_simulation: 1728000 result_base_name: oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0_r1129607049 The child process has been launched with process id: 84 Executing the command: /home/seb/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:84:84] [20240621:021317:1718928797:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 02:13:27 STEP 480 H= 120:00 +CPU= 8.305 02:13:35 STEP 481 H= 120:15 +CPU= 7.597 02:13:42 STEP 482 H= 120:30 +CPU= 7.500 02:13:50 STEP 483 H= 120:45 +CPU= 7.499 02:13:58 STEP 484 H= 121:00 +CPU= 7.765 02:14:06 STEP 485 H= 121:15 +CPU= 7.943 02:14:14 STEP 486 H= 121:30 +CPU= 8.018 02:14:22 STEP 487 H= 121:45 +CPU= 8.006 02:14:30 STEP 488 H= 122:00 +CPU= 8.186 02:14:38 STEP 489 H= 122:15 +CPU= 7.810 02:14:46 STEP 490 H= 122:30 +CPU= 7.856 02:14:54 STEP 491 H= 122:45 +CPU= 7.920 02:15:04 STEP 492 H= 123:00 +CPU= 10.101 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000492 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000492 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000492 02:15:12 STEP 493 H= 123:15 +CPU= 7.846 02:15:19 STEP 494 H= 123:30 +CPU= 7.735 02:15:27 STEP 495 H= 123:45 +CPU= 7.781 02:15:35 STEP 496 H= 124:00 +CPU= 7.939 02:15:43 STEP 497 H= 124:15 +CPU= 7.677 02:15:51 STEP 498 H= 124:30 +CPU= 7.711 02:15:58 STEP 499 H= 124:45 +CPU= 7.672 02:16:06 STEP 500 H= 125:00 +CPU= 7.659 02:16:14 STEP 501 H= 125:15 +CPU= 7.415 02:16:21 STEP 502 H= 125:30 +CPU= 7.526 02:16:29 STEP 503 H= 125:45 +CPU= 7.490 02:16:38 STEP 504 H= 126:00 +CPU= 9.547 02:16:46 STEP 505 H= 126:15 +CPU= 7.565 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000504 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000504 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000504 02:16:53 STEP 506 H= 126:30 +CPU= 7.629 02:17:01 STEP 507 H= 126:45 +CPU= 7.943 02:17:09 STEP 508 H= 127:00 +CPU= 7.794 02:17:17 STEP 509 H= 127:15 +CPU= 7.816 02:17:25 STEP 510 H= 127:30 +CPU= 7.834 02:17:33 STEP 511 H= 127:45 +CPU= 7.878 02:17:41 STEP 512 H= 128:00 +CPU= 8.040 02:17:49 STEP 513 H= 128:15 +CPU= 7.842 02:17:57 STEP 514 H= 128:30 +CPU= 7.783 02:18:05 STEP 515 H= 128:45 +CPU= 7.837 02:18:15 STEP 516 H= 129:00 +CPU= 9.900 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000516 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000516 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000516 02:18:22 STEP 517 H= 129:15 +CPU= 7.856 02:18:30 STEP 518 H= 129:30 +CPU= 7.884 02:18:38 STEP 519 H= 129:45 +CPU= 7.889 02:18:46 STEP 520 H= 130:00 +CPU= 8.057 02:18:54 STEP 521 H= 130:15 +CPU= 7.871 02:19:02 STEP 522 H= 130:30 +CPU= 7.905 02:19:10 STEP 523 H= 130:45 +CPU= 7.831 02:19:18 STEP 524 H= 131:00 +CPU= 8.130 02:19:26 STEP 525 H= 131:15 +CPU= 8.031 02:19:34 STEP 526 H= 131:30 +CPU= 7.942 02:19:42 STEP 527 H= 131:45 +CPU= 7.808 02:19:54 STEP 528 H= 132:00 +CPU= 11.106 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000528 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000528 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000528 02:20:01 STEP 529 H= 132:15 +CPU= 7.698 02:20:09 STEP 530 H= 132:30 +CPU= 7.771 02:20:17 STEP 531 H= 132:45 +CPU= 7.927 02:20:25 STEP 532 H= 133:00 +CPU= 8.165 02:20:33 STEP 533 H= 133:15 +CPU= 8.031 02:20:41 STEP 534 H= 133:30 +CPU= 7.564 02:20:49 STEP 535 H= 133:45 +CPU= 7.549 02:20:56 STEP 536 H= 134:00 +CPU= 7.688 02:21:04 STEP 537 H= 134:15 +CPU= 7.587 02:21:12 STEP 538 H= 134:30 +CPU= 7.790 02:21:20 STEP 539 H= 134:45 +CPU= 7.877 02:21:30 STEP 540 H= 135:00 +CPU= 9.870 02:21:37 STEP 541 H= 135:15 +CPU= 7.544 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000540 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000540 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000540 02:21:45 STEP 542 H= 135:30 +CPU= 7.622 02:21:52 STEP 543 H= 135:45 +CPU= 7.648 02:22:00 STEP 544 H= 136:00 +CPU= 7.781 02:22:08 STEP 545 H= 136:15 +CPU= 7.674 02:22:15 STEP 546 H= 136:30 +CPU= 7.499 02:22:23 STEP 547 H= 136:45 +CPU= 7.644 02:22:31 STEP 548 H= 137:00 +CPU= 8.152 02:22:39 STEP 549 H= 137:15 +CPU= 7.783 02:22:47 STEP 550 H= 137:30 +CPU= 7.946 02:22:55 STEP 551 H= 137:45 +CPU= 7.682 02:23:05 STEP 552 H= 138:00 +CPU= 10.082 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000552 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000552 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000552 02:23:13 STEP 553 H= 138:15 +CPU= 7.574 02:23:20 STEP 554 H= 138:30 +CPU= 7.371 02:23:28 STEP 555 H= 138:45 +CPU= 7.517 02:23:36 STEP 556 H= 139:00 +CPU= 8.062 02:23:43 STEP 557 H= 139:15 +CPU= 7.400 02:23:51 STEP 558 H= 139:30 +CPU= 7.683 02:23:58 STEP 559 H= 139:45 +CPU= 7.662 02:24:06 STEP 560 H= 140:00 +CPU= 7.718 02:24:14 STEP 561 H= 140:15 +CPU= 7.820 02:24:22 STEP 562 H= 140:30 +CPU= 7.666 02:24:29 STEP 563 H= 140:45 +CPU= 7.415 02:24:39 STEP 564 H= 141:00 +CPU= 9.960 02:24:47 STEP 565 H= 141:15 +CPU= 7.833 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000564 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000564 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000564 02:24:55 STEP 566 H= 141:30 +CPU= 7.999 02:25:03 STEP 567 H= 141:45 +CPU= 7.838 02:25:11 STEP 568 H= 142:00 +CPU= 7.726 02:25:18 STEP 569 H= 142:15 +CPU= 7.605 02:25:26 STEP 570 H= 142:30 +CPU= 7.476 02:25:33 STEP 571 H= 142:45 +CPU= 7.620 02:25:42 STEP 572 H= 143:00 +CPU= 8.059 02:25:49 STEP 573 H= 143:15 +CPU= 7.732 02:25:57 STEP 574 H= 143:30 +CPU= 7.573 02:26:04 STEP 575 H= 143:45 +CPU= 7.534 02:26:15 STEP 576 H= 144:00 +CPU= 10.925 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000576 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000576 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000576 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000480 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000480 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000480 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000492 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000492 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000492 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000504 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000504 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000504 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000516 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000516 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000516 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000528 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000528 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000528 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000540 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000540 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000540 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000552 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000552 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000552 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMGGh7zg+000564 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMSHh7zg+000564 Adding to the zip: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796/ICMUAh7zg+000564 Zipping up the intermediate file: /home/seb/projects/climateprediction.net/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0_r1129607049_5.zip Uploading the intermediate file: upload_file_5.zip 02:26:24 STEP 577 H= 144:15 +CPU= 8.225 02:26:32 STEP 578 H= 144:30 +CPU= 8.097 02:26:40 STEP 579 H= 144:45 +CPU= 7.921 02:26:48 STEP 580 H= 145:00 +CPU= 8.138 02:26:55 STEP 581 H= 145:15 +CPU= 7.452 02:27:03 STEP 582 H= 145:30 +CPU= 7.804 02:27:11 STEP 583 H= 145:45 +CPU= 7.582 02:27:19 STEP 584 H= 146:00 +CPU= 7.825 02:27:26 STEP 585 H= 146:15 +CPU= 7.688 02:27:34 STEP 586 H= 146:30 +CPU= 7.484 02:27:42 STEP 587 H= 146:45 +CPU= 7.590 02:27:52 STEP 588 H= 147:00 +CPU= 10.074 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000588 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000588 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000588 02:28:00 STEP 589 H= 147:15 +CPU= 7.882 02:28:08 STEP 590 H= 147:30 +CPU= 7.979 02:28:16 STEP 591 H= 147:45 +CPU= 7.872 02:28:24 STEP 592 H= 148:00 +CPU= 8.065 02:28:32 STEP 593 H= 148:15 +CPU= 7.942 02:28:40 STEP 594 H= 148:30 +CPU= 7.948 02:28:48 STEP 595 H= 148:45 +CPU= 7.994 02:28:56 STEP 596 H= 149:00 +CPU= 8.151 02:29:04 STEP 597 H= 149:15 +CPU= 8.060 02:29:12 STEP 598 H= 149:30 +CPU= 7.892 02:29:20 STEP 599 H= 149:45 +CPU= 7.756 02:29:30 STEP 600 H= 150:00 +CPU= 10.066 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000600 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000600 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000600 02:29:38 STEP 601 H= 150:15 +CPU= 7.839 02:29:46 STEP 602 H= 150:30 +CPU= 7.905 02:29:54 STEP 603 H= 150:45 +CPU= 7.824 02:30:02 STEP 604 H= 151:00 +CPU= 8.010 02:30:10 STEP 605 H= 151:15 +CPU= 7.850 02:30:17 STEP 606 H= 151:30 +CPU= 7.625 02:30:25 STEP 607 H= 151:45 +CPU= 7.655 02:30:33 STEP 608 H= 152:00 +CPU= 7.872 02:30:41 STEP 609 H= 152:15 +CPU= 7.763 02:30:49 STEP 610 H= 152:30 +CPU= 7.868 02:30:57 STEP 611 H= 152:45 +CPU= 7.785 02:31:07 STEP 612 H= 153:00 +CPU= 10.017 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000612 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000612 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000612 02:31:15 STEP 613 H= 153:15 +CPU= 7.944 02:31:23 STEP 614 H= 153:30 +CPU= 8.074 02:31:31 STEP 615 H= 153:45 +CPU= 7.901 02:31:39 STEP 616 H= 154:00 +CPU= 8.095 02:31:47 STEP 617 H= 154:15 +CPU= 7.792 02:31:55 STEP 618 H= 154:30 +CPU= 7.876 02:32:03 STEP 619 H= 154:45 +CPU= 7.911 02:32:11 STEP 620 H= 155:00 +CPU= 8.111 02:32:19 STEP 621 H= 155:15 +CPU= 8.124 02:32:27 STEP 622 H= 155:30 +CPU= 7.809 02:32:35 STEP 623 H= 155:45 +CPU= 7.777 02:32:46 STEP 624 H= 156:00 +CPU= 11.113 Moving to projects directory: /home/seb/slots/0/ICMGGh7zg+000624 Moving to projects directory: /home/seb/slots/0/ICMSHh7zg+000624 Moving to projects directory: /home/seb/slots/0/ICMUAh7zg+000624 02:32:54 STEP 625 H= 156:15 +CPU= 7.940 02:33:02 STEP 626 H= 156:30 +CPU= 7.959 02:33:10 STEP 627 H= 156:45 +CPU= 7.954 02:33:18 STEP 628 H= 157:00 +CPU= 8.157 02:33:26 STEP 629 H= 157:15 +CPU= 7.938 02:33:34 STEP 630 H= 157:30 +CPU= 7.913 02:33:42 STEP 631 H= 157:45 +CPU= 7.902 02:33:50 STEP 632 H= 158:00 +CPU= 7.976 02:33:56 (78): BOINC client no longer exists - exiting 02:33:56 (78): timer handler: client dead, exiting No heartbeat received from BOINC client, ending the child process 02:34:01 (78): BOINC client no longer exists - exiting 02:34:01 (78): timer handler: client dead, exiting 02:34:06 (78): BOINC client no longer exists - exiting 02:34:06 (78): timer handler: client dead, exiting 02:34:11 (78): BOINC client no longer exists - exiting 02:34:11 (78): timer handler: client dead, exiting 02:34:16 (78): BOINC client no longer exists - exiting 02:34:16 (78): timer handler: client dead, exiting 02:34:21 (78): BOINC client no longer exists - exiting 02:34:21 (78): timer handler: client dead, exiting 02:34:26 (78): BOINC client no longer exists - exiting 02:34:26 (78): timer handler: client dead, exiting 02:34:31 (78): BOINC client no longer exists - exiting 02:34:31 (78): timer handler: client dead, exiting 02:34:36 (78): BOINC client no longer exists - exiting 02:34:36 (78): timer handler: client dead, exiting 02:34:41 (78): BOINC client no longer exists - exiting 02:34:41 (78): timer handler: client dead, exiting 02:34:46 (78): BOINC client no longer exists - exiting 02:34:46 (78): timer handler: client dead, exiting 02:34:51 (78): BOINC client no longer exists - exiting 02:34:51 (78): timer handler: client dead, exiting 02:34:54 (131): Can't acquire lockfile (-154) - waiting 35s 02:34:56(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a1wl (argv4) batchid: 1018 (argv5) wuid: 12290796 (argv6) fclen: 20 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /home/seb/slots/0 Project directory is: /home/seb/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.13 Location of temp folder: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796 ..mkdir for temp folder for results failed Copying: /home/seb/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/seb/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /home/seb/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_6c71865a3882b3b62f5c7f2394aa73be to: /home/seb/slots/0/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796.zip Unzipping the namelist zip file: /home/seb/slots/0/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796.zip ic_ancil_file: ic_ancil_12290796 ifsdata_file: ifsdata_12290796 climate_data_file: clim_data_12290796 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/seb/slots/0/ic_ancil_12290796.zip Unzipping the IC ancils zip file: /home/seb/slots/0/ic_ancil_12290796.zip ..mkdir for ifsdata folder failed Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5a377634e1417298b1132a8f0e110fb8 to: /home/seb/slots/0/ifsdata/ifsdata_12290796.zip Unzipping the ifsdata_zip file: /home/seb/slots/0/ifsdata/ifsdata_12290796.zip ..mkdir for the climate data folder failed Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/seb/slots/0/159l_2/clim_data_12290796.zip Unzipping the climate data zip file: /home/seb/slots/0/159l_2/clim_data_12290796.zip Checking for progress XML file: /home/seb/slots/0/progress_file_12290796.xml Opened progress file ok : /home/seb/slots/0/progress_file_12290796.xml -- Model is restarting -- Adjusting last_iter, 633, to previous model restart step. Creating progress file: /home/seb/slots/0/progress_file_12290796.xml last_cpu_time: 5897 upload_file_number: 6 last_iter: 625 last_upload: 518400 model_completed: 0 total_length_of_simulation: 1728000 result_base_name: oifs_43r3_bl_a1wl_2016092300_20_1018_12290796_0_r1129607049 The child process has been launched with process id: 179 Executing the command: /home/seb/slots/0/oifs_43r3_model.exe [EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef> [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef> [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1 [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil)) [EC_DRHOOK:seb-station-manjaro:1:1:179:179] [20240621:023531:1718930131:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef> 02:35:34 (131): BOINC client no longer exists - exiting 02:35:34 (131): timer handler: client dead, exiting No heartbeat received from BOINC client, ending the child process 02:35:39 (131): BOINC client no longer exists - exiting 02:35:39 (131): timer handler: client dead, exiting 02:35:44 (131): BOINC client no longer exists - exiting 02:35:44 (131): timer handler: client dead, exiting 02:35:49 (131): BOINC client no longer exists - exiting 02:35:49 (131): timer handler: client dead, exiting 02:35:54 (131): BOINC client no longer exists - exiting 02:35:54 (131): timer handler: client dead, exiting 02:35:59 (131): BOINC client no longer exists - exiting 02:35:59 (131): timer handler: client dead, exiting 02:36:04 (131): BOINC client no longer exists - exiting 02:36:04 (131): timer handler: client dead, exiting 02:36:09 (131): BOINC client no longer exists - exiting 02:36:09 (131): timer handler: client dead, exiting 02:36:14 (131): BOINC client no longer exists - exiting 02:36:14 (131): timer handler: client dead, exiting 02:36:19 (131): BOINC client no longer exists - exiting 02:36:19 (131): timer handler: client dead, exiting 02:36:24 (131): BOINC client no longer exists - exiting 02:36:24 (131): timer handler: client dead, exiting 02:36:29 (131): BOINC client no longer exists - exiting 02:36:29 (131): timer handler: client dead, exiting 02:36:34 (131): BOINC client no longer exists - exiting 02:36:34 (131): timer handler: client dead, exiting >>> Printing last 70 lines from file: NODE.001_01 Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Black_Carbon_hydrophobic(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Organic_Matter_hydrophilic(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Organic_Matter_hydrophobic(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Mineral_Dust_bin1(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Mineral_Dust_bin2(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Mineral_Dust_bin3(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sea_Salt_bin1(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sea_Salt_bin2(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sea_Salt_bin3(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc Reading Sulfates(120,61,12) Closing NetCDF file ifsdata/aerosol_cams_climatology_43R3a.nc UPDTIM MACC AEROSOL CLIMATOLOGY IS LOADED SUECAEC: INLEV,ISYEAR,IEYEAR= 0 1957 2010 suecaec NAERMACC = 1 UPDT RAESHDU = 2000.0 SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20160923 810 9.999999999999998E-005 NORMS AT NSTEP CNT4 624 SPECTRAL NORMS - LOG(PREHYDS) 0.115260884479326E+02 OROGRAPHY 0.000000000000000E+00 LEV VORTICITY DIVERGENCE TEMPERATURE KINETIC ENERGY AVE 0.218458904368848E-05 0.127509553642404E-07 0.194859079902964E+03 0.784039396275479E+01 GPNORM HUMIDITY AVERAGE MINIMUM MAXIMUM AVE 0.341844864744162E-03 0.999999999999999E-08 0.423371298430283E-02 GPNORM LIQUID WATER AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM ICE WATER AVERAGE MINIMUM MAXIMUM AVE 0.270146518718220E-07 0.000000000000000E+00 0.180676854838896E-06 GPNORM SNOW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM RAIN AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CLOUD FRACTION AVERAGE MINIMUM MAXIMUM AVE 0.154063954693155E+00 0.000000000000000E+00 0.100000000000000E+01 GPNORM CLW AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CIW AVERAGE MINIMUM MAXIMUM AVE 0.270126318148835E-07 0.000000000000000E+00 0.180674879452525E-06 GPNORM CC AVERAGE MINIMUM MAXIMUM AVE 0.154091367936855E+00 0.000000000000000E+00 0.100000000000000E+01 GPNORM RFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM SFL AVERAGE MINIMUM MAXIMUM AVE 0.143447514550973E-09 0.000000000000000E+00 0.139507155168258E-08 GPNORM CRFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM CSFL AVERAGE MINIMUM MAXIMUM AVE 0.000000000000000E+00 0.000000000000000E+00 0.000000000000000E+00 GPNORM PFRC AVERAGE MINIMUM MAXIMUM AVE 0.221840987869860E-01 0.000000000000000E+00 0.100000000000000E+01 NSTEP = 624 STEPO 0AAA00AAA ------------------------------------------------ oifs_get_stat: Error. ifs.stat file is not open CNT0 not found; string returned was: '' >>> Printing last 2 lines from file: ifs.stat 02:35:31 000000000 CNT3 -999 0.174 0.174 0.174 0:00 0:00 0.00000000000000E+00 1GB 0MB 02:35:32 0AAA00AAA STEPO 624 1.223 1.223 1.224 0:01 0:01 0.12750955364240E-07 1GB 0MB ------------------------------------------------ >>> Printing last 11 lines from file: rcf CSTEP = " 624", CTIME = "00061200 ", NSTEPLPP = 201, 202, 49, 228226, 228227, 2*0, 612, 2*0, IPRGPNSRES = 1, IPRGPEWRES = 1, IPRTRWRES = 1, IPRTRVRES = 1, GMASS0 = 0.000000000000000E+000, GMASSI = 0.000000000000000E+000 / ------------------------------------------------ ..Failed, model did not complete successfully (argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu (argv1) start_date: 2016092300 (argv2) exptid: h7zg (argv3) unique_member_id: a1wl (argv4) batchid: 1018 (argv5) wuid: 12290796 (argv6) fclen: 20 (argv7) app_name: oifs_43r3_bl (argv8) nthreads: 1 Working directory is: /home/seb/slots/0 Project directory is: /home/seb/projects/climateprediction.net/ app name: oifs_43r3_bl version: 1.13 Location of temp folder: /home/seb/projects/climateprediction.net/oifs_43r3_bl_12290796 ..mkdir for temp folder for results failed Copying: /home/seb/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /home/seb/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Unzipping the app zip file: /home/seb/slots/0/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip Copying the namelist files from: ../../projects/climateprediction.net/jf_6c71865a3882b3b62f5c7f2394aa73be to: /home/seb/slots/0/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796.zip Unzipping the namelist zip file: /home/seb/slots/0/oifs_43r3_bl_a1wl_2016092300_20_1018_12290796.zip ic_ancil_file: ic_ancil_12290796 ifsdata_file: ifsdata_12290796 climate_data_file: clim_data_12290796 horiz_resolution: 159 vert_resolution: 91 grid_type: l_2 upload_interval: 96 utstep: 900 nfrres: restart dump frequency (steps) 48 Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /home/seb/slots/0/ic_ancil_12290796.zip ..Copying the IC ancils to the working directory failed </stderr_txt> ]]> |
No trickles! |
---|
©2025 cpdn.org