climateprediction.net home page
Task 22437622

Task 22437622

Name oifs_43r3_bl_a385_2016092300_20_1017_12286008_0
Workunit 12286008
Created 7 Jun 2024, 16:53:27 UTC
Sent 8 Jun 2024, 8:42:04 UTC
Report deadline 7 Aug 2024, 8:42:04 UTC
Received 9 Jun 2024, 2:35:36 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1494452
Run time 7 hours 56 min 4 sec
CPU time 6 hours 52 min 48 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 7.32 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,555.47 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,351.66 MB
Stderr 
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
H= 314:45 +CPU= 17.029
  15:29:06 STEP 1260 H= 315:00 +CPU= 21.796
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001260
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001260
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001260
  15:29:34 STEP 1261 H= 315:15 +CPU= 16.925
  15:30:03 STEP 1262 H= 315:30 +CPU= 16.979
  15:30:32 STEP 1263 H= 315:45 +CPU= 17.081
  15:31:00 STEP 1264 H= 316:00 +CPU= 17.370
  15:31:29 STEP 1265 H= 316:15 +CPU= 16.973
  15:31:58 STEP 1266 H= 316:30 +CPU= 17.226
  15:32:26 STEP 1267 H= 316:45 +CPU= 17.235
  15:32:57 STEP 1268 H= 317:00 +CPU= 17.702
  15:33:26 STEP 1269 H= 317:15 +CPU= 17.154
  15:33:55 STEP 1270 H= 317:30 +CPU= 17.198
  15:34:25 STEP 1271 H= 317:45 +CPU= 17.021
  15:35:02 STEP 1272 H= 318:00 +CPU= 22.043
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001272
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001272
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001272
  15:35:32 STEP 1273 H= 318:15 +CPU= 17.130
  15:36:02 STEP 1274 H= 318:30 +CPU= 17.239
  15:36:31 STEP 1275 H= 318:45 +CPU= 17.269
  15:37:02 STEP 1276 H= 319:00 +CPU= 17.846
  15:37:26 STEP 1277 H= 319:15 +CPU= 16.704
  15:37:43 STEP 1278 H= 319:30 +CPU= 15.584
  15:38:09 STEP 1279 H= 319:45 +CPU= 16.790
  15:38:36 STEP 1280 H= 320:00 +CPU= 17.235
  15:39:03 STEP 1281 H= 320:15 +CPU= 16.699
  15:39:30 STEP 1282 H= 320:30 +CPU= 16.866
  15:39:56 STEP 1283 H= 320:45 +CPU= 16.493
  15:40:31 STEP 1284 H= 321:00 +CPU= 21.411
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001284
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001284
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001284
  15:40:58 STEP 1285 H= 321:15 +CPU= 16.793
  15:41:24 STEP 1286 H= 321:30 +CPU= 16.768
  15:41:51 STEP 1287 H= 321:45 +CPU= 16.843
  15:42:19 STEP 1288 H= 322:00 +CPU= 17.288
  15:42:47 STEP 1289 H= 322:15 +CPU= 16.907
  15:43:14 STEP 1290 H= 322:30 +CPU= 16.773
  15:43:42 STEP 1291 H= 322:45 +CPU= 16.896
  15:44:10 STEP 1292 H= 323:00 +CPU= 17.144
  15:44:37 STEP 1293 H= 323:15 +CPU= 16.902
  15:45:05 STEP 1294 H= 323:30 +CPU= 16.990
  15:45:33 STEP 1295 H= 323:45 +CPU= 16.878
  15:46:12 STEP 1296 H= 324:00 +CPU= 24.164
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001296
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001296
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001296
  15:46:39 STEP 1297 H= 324:15 +CPU= 16.635
  15:47:06 STEP 1298 H= 324:30 +CPU= 16.697
  15:47:32 STEP 1299 H= 324:45 +CPU= 16.608
  15:48:02 STEP 1300 H= 325:00 +CPU= 17.382
  15:48:30 STEP 1301 H= 325:15 +CPU= 16.876
  15:48:58 STEP 1302 H= 325:30 +CPU= 16.908
  15:49:26 STEP 1303 H= 325:45 +CPU= 16.804
  15:49:54 STEP 1304 H= 326:00 +CPU= 17.569
  15:50:22 STEP 1305 H= 326:15 +CPU= 16.857
  15:50:50 STEP 1306 H= 326:30 +CPU= 17.153
  15:51:17 STEP 1307 H= 326:45 +CPU= 16.867
  15:51:54 STEP 1308 H= 327:00 +CPU= 22.087
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001308
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001308
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001308
  15:52:22 STEP 1309 H= 327:15 +CPU= 17.252
  15:52:49 STEP 1310 H= 327:30 +CPU= 16.641
  15:53:15 STEP 1311 H= 327:45 +CPU= 16.627
  15:53:43 STEP 1312 H= 328:00 +CPU= 17.158
  15:54:10 STEP 1313 H= 328:15 +CPU= 16.638
  15:54:37 STEP 1314 H= 328:30 +CPU= 16.796
  15:55:04 STEP 1315 H= 328:45 +CPU= 16.858
  15:55:32 STEP 1316 H= 329:00 +CPU= 17.214
  15:56:00 STEP 1317 H= 329:15 +CPU= 16.850
  15:56:27 STEP 1318 H= 329:30 +CPU= 16.745
  15:56:54 STEP 1319 H= 329:45 +CPU= 16.610
  15:57:29 STEP 1320 H= 330:00 +CPU= 21.540
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001320
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001320
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001320
  15:57:56 STEP 1321 H= 330:15 +CPU= 16.919
  15:58:24 STEP 1322 H= 330:30 +CPU= 16.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  16:51:31 STEP 1323 H= 330:45 +CPU= 17.030
  16:52:01 STEP 1324 H= 331:00 +CPU= 17.585
  16:52:30 STEP 1325 H= 331:15 +CPU= 17.029
  16:53:00 STEP 1326 H= 331:30 +CPU= 17.071
  16:53:29 STEP 1327 H= 331:45 +CPU= 16.871
  16:53:58 STEP 1328 H= 332:00 +CPU= 17.408
  16:54:25 STEP 1329 H= 332:15 +CPU= 16.992
  16:54:54 STEP 1330 H= 332:30 +CPU= 16.873
  16:55:22 STEP 1331 H= 332:45 +CPU= 16.871
  16:55:59 STEP 1332 H= 333:00 +CPU= 21.773
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001332
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001332
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001332
  16:56:28 STEP 1333 H= 333:15 +CPU= 17.100
  16:56:57 STEP 1334 H= 333:30 +CPU= 17.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:11:38 STEP 1335 H= 333:45 +CPU= 17.127
  17:12:06 STEP 1336 H= 334:00 +CPU= 17.772
  17:12:33 STEP 1337 H= 334:15 +CPU= 16.912
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:25:24 STEP 1338 H= 334:30 +CPU= 16.852
  17:25:51 STEP 1339 H= 334:45 +CPU= 16.982
  17:26:20 STEP 1340 H= 335:00 +CPU= 17.563
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:29:35 STEP 1341 H= 335:15 +CPU= 16.993
  17:30:03 STEP 1342 H= 335:30 +CPU= 17.256
  17:30:31 STEP 1343 H= 335:45 +CPU= 16.978
  17:31:11 STEP 1344 H= 336:00 +CPU= 24.028
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001344
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001344
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001248
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001248
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001248
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001260
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001260
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001260
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001272
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001272
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001272
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001284
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001284
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001284
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001296
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001296
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001296
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001308
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001308
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001308
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001320
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001320
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001320
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001332
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001332
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001332
Zipping up the intermediate file: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_13.zip
Uploading the intermediate file: upload_file_13.zip
  17:31:39 STEP 1345 H= 336:15 +CPU= 16.903
  17:32:06 STEP 1346 H= 336:30 +CPU= 16.792
  17:32:34 STEP 1347 H= 336:45 +CPU= 17.257
  17:33:03 STEP 1348 H= 337:00 +CPU= 17.503
  17:33:31 STEP 1349 H= 337:15 +CPU= 17.115
  17:33:59 STEP 1350 H= 337:30 +CPU= 17.081
  17:34:26 STEP 1351 H= 337:45 +CPU= 16.897
  17:34:55 STEP 1352 H= 338:00 +CPU= 17.464
  17:35:21 STEP 1353 H= 338:15 +CPU= 16.737
  17:35:50 STEP 1354 H= 338:30 +CPU= 17.153
  17:36:18 STEP 1355 H= 338:45 +CPU= 17.065
  17:36:55 STEP 1356 H= 339:00 +CPU= 22.103
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001356
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001356
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001356
  17:37:22 STEP 1357 H= 339:15 +CPU= 17.139
  17:37:51 STEP 1358 H= 339:30 +CPU= 17.015
  17:38:19 STEP 1359 H= 339:45 +CPU= 17.184
  17:38:48 STEP 1360 H= 340:00 +CPU= 17.693
  17:39:16 STEP 1361 H= 340:15 +CPU= 17.290
  17:39:43 STEP 1362 H= 340:30 +CPU= 17.180
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:44:42 STEP 1363 H= 340:45 +CPU= 17.070
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:47:06 STEP 1364 H= 341:00 +CPU= 17.269
  17:47:34 STEP 1365 H= 341:15 +CPU= 16.586
  17:48:02 STEP 1366 H= 341:30 +CPU= 16.824
  17:48:31 STEP 1367 H= 341:45 +CPU= 16.831
  17:49:07 STEP 1368 H= 342:00 +CPU= 21.779
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001368
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001368
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001368
  17:49:34 STEP 1369 H= 342:15 +CPU= 17.086
  17:50:03 STEP 1370 H= 342:30 +CPU= 17.380
  17:50:32 STEP 1371 H= 342:45 +CPU= 17.545
  17:51:01 STEP 1372 H= 343:00 +CPU= 17.720
  17:51:30 STEP 1373 H= 343:15 +CPU= 17.186
  17:51:59 STEP 1374 H= 343:30 +CPU= 17.251
  17:52:28 STEP 1375 H= 343:45 +CPU= 17.453
  17:52:57 STEP 1376 H= 344:00 +CPU= 17.845
  17:53:25 STEP 1377 H= 344:15 +CPU= 17.181
  17:53:53 STEP 1378 H= 344:30 +CPU= 17.248
  17:54:21 STEP 1379 H= 344:45 +CPU= 16.892
  17:54:57 STEP 1380 H= 345:00 +CPU= 21.695
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001380
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001380
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001380
  17:55:24 STEP 1381 H= 345:15 +CPU= 16.749
  17:55:52 STEP 1382 H= 345:30 +CPU= 16.850
  17:56:19 STEP 1383 H= 345:45 +CPU= 16.598
  17:56:46 STEP 1384 H= 346:00 +CPU= 17.208
  17:57:13 STEP 1385 H= 346:15 +CPU= 16.922
  17:57:42 STEP 1386 H= 346:30 +CPU= 16.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:20:31 STEP 1387 H= 346:45 +CPU= 16.663
  18:20:48 STEP 1388 H= 347:00 +CPU= 16.315
  18:21:04 STEP 1389 H= 347:15 +CPU= 16.392
  18:21:21 STEP 1390 H= 347:30 +CPU= 15.991
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:47:16 STEP 1391 H= 347:45 +CPU= 16.455
  20:47:39 STEP 1392 H= 348:00 +CPU= 23.081
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001392
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001392
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001392
  20:47:55 STEP 1393 H= 348:15 +CPU= 16.117
  20:48:11 STEP 1394 H= 348:30 +CPU= 15.678
  20:48:27 STEP 1395 H= 348:45 +CPU= 15.912
  20:48:43 STEP 1396 H= 349:00 +CPU= 16.144
  20:48:58 STEP 1397 H= 349:15 +CPU= 15.487
  20:49:14 STEP 1398 H= 349:30 +CPU= 16.151
  20:49:30 STEP 1399 H= 349:45 +CPU= 15.611
  20:49:46 STEP 1400 H= 350:00 +CPU= 16.147
  20:50:03 STEP 1401 H= 350:15 +CPU= 16.252
  20:50:19 STEP 1402 H= 350:30 +CPU= 16.479
  20:50:35 STEP 1403 H= 350:45 +CPU= 15.825
  20:50:57 STEP 1404 H= 351:00 +CPU= 21.460
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001404
  20:51:14 STEP 1405 H= 351:15 +CPU= 16.350
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:53:47 STEP 1406 H= 351:30 +CPU= 15.854
  20:54:04 STEP 1407 H= 351:45 +CPU= 15.998
  20:54:20 STEP 1408 H= 352:00 +CPU= 16.053
  20:54:36 STEP 1409 H= 352:15 +CPU= 15.727
  20:54:52 STEP 1410 H= 352:30 +CPU= 15.595
  20:55:07 STEP 1411 H= 352:45 +CPU= 15.401
  20:55:23 STEP 1412 H= 353:00 +CPU= 15.837
  20:55:39 STEP 1413 H= 353:15 +CPU= 16.317
  20:55:55 STEP 1414 H= 353:30 +CPU= 15.384
  20:56:11 STEP 1415 H= 353:45 +CPU= 16.042
  20:56:32 STEP 1416 H= 354:00 +CPU= 20.487
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001416
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001416
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001416
  20:56:49 STEP 1417 H= 354:15 +CPU= 16.339
  20:57:05 STEP 1418 H= 354:30 +CPU= 15.908
  20:57:21 STEP 1419 H= 354:45 +CPU= 15.841
  20:57:37 STEP 1420 H= 355:00 +CPU= 15.916
  20:57:53 STEP 1421 H= 355:15 +CPU= 16.310
  20:58:09 STEP 1422 H= 355:30 +CPU= 15.559
  20:58:25 STEP 1423 H= 355:45 +CPU= 15.654
  20:58:42 STEP 1424 H= 356:00 +CPU= 17.078
  20:58:58 STEP 1425 H= 356:15 +CPU= 15.669
  20:59:14 STEP 1426 H= 356:30 +CPU= 15.989
  20:59:30 STEP 1427 H= 356:45 +CPU= 15.779
  20:59:51 STEP 1428 H= 357:00 +CPU= 20.580
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001428
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001428
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001428
  21:00:07 STEP 1429 H= 357:15 +CPU= 15.871
  21:00:22 STEP 1430 H= 357:30 +CPU= 14.983
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.456] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 5751MB (heap), 5565MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.468] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.468] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.469] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.469] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.469] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 5751 MB (maxheap), 5565 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     2        (1005) 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               LAPINEB 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                LARCINB 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 LAITRE_GMV 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210028:1717894828:58632.499] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                  LAITRI 
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210029:1717894829:58633.500] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:185892:185892] [20240608:210029:1717894829:58633.500] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DCADA8  laitri_                   352  laitri.F90
oifs_43r3_model.e  0000000000DC6EE4  laitre_gmv_               143  laitre_gmv.F90
oifs_43r3_model.e  0000000000DCE6AB  larcinb_                  301  larcinb.F90
oifs_43r3_model.e  0000000000D8A8F8  lapineb_                  451  lapineb.F90
oifs_43r3_model.e  0000000000D57FC1  call_sl_                  363  call_sl.F90
oifs_43r3_model.e  0000000000D54BBC  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BD4FD8  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a385
(argv4) batchid: 1017
(argv5) wuid: 12286008
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /more/var/lib/boinc-client/slots/16
Project directory is: /more/var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008
..mkdir for temp folder for results failed
Copying: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a9954dfb891619eed0961f1057f59837 to: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_a385_2016092300_20_1017_12286008.zip
Unzipping the namelist zip file: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_a385_2016092300_20_1017_12286008.zip
ic_ancil_file: ic_ancil_12286008
ifsdata_file: ifsdata_12286008
climate_data_file: clim_data_12286008
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /more/var/lib/boinc-client/slots/16/ic_ancil_12286008.zip
Unzipping the IC ancils zip file: /more/var/lib/boinc-client/slots/16/ic_ancil_12286008.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_411beeabc3b5df374313efcd0cb5b8ec to: /more/var/lib/boinc-client/slots/16/ifsdata/ifsdata_12286008.zip
Unzipping the ifsdata_zip file: /more/var/lib/boinc-client/slots/16/ifsdata/ifsdata_12286008.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /more/var/lib/boinc-client/slots/16/159l_2/clim_data_12286008.zip
Unzipping the climate data zip file: /more/var/lib/boinc-client/slots/16/159l_2/clim_data_12286008.zip
Checking for progress XML file: /more/var/lib/boinc-client/slots/16/progress_file_12286008.xml
Opened progress file ok : /more/var/lib/boinc-client/slots/16/progress_file_12286008.xml
-- Model is restarting --
Adjusting last_iter, 1431, to previous model restart step.
Creating progress file: /more/var/lib/boinc-client/slots/16/progress_file_12286008.xml
last_cpu_time: 23735
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772
The child process has been launched with process id: 214054
Executing the command: /more/var/lib/boinc-client/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210127:1717894887:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:01:46 STEP 1392 H= 348:00 +CPU= 16.341
  21:02:00 STEP 1393 H= 348:15 +CPU= 14.260
  21:02:15 STEP 1394 H= 348:30 +CPU= 15.149
  21:02:32 STEP 1395 H= 348:45 +CPU= 16.302
  21:02:49 STEP 1396 H= 349:00 +CPU= 17.288
  21:03:06 STEP 1397 H= 349:15 +CPU= 16.410
  21:03:21 STEP 1398 H= 349:30 +CPU= 15.730
  21:03:37 STEP 1399 H= 349:45 +CPU= 15.795
  21:03:53 STEP 1400 H= 350:00 +CPU= 15.769
  21:04:08 STEP 1401 H= 350:15 +CPU= 15.619
  21:04:24 STEP 1402 H= 350:30 +CPU= 16.041
  21:04:40 STEP 1403 H= 350:45 +CPU= 15.991
  21:05:02 STEP 1404 H= 351:00 +CPU= 21.151
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001404
Suspend request received from the BOINC client, suspending the child process
double free or corruption (top)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3958MB (heap), 5315MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 5315 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :               GPGEO 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                VERINT 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210526:1717895126:239.407] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] :                 VERINT_DGEMM_2 
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210527:1717895127:240.407] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:214054:214054] [20240608:210527:1717895127:240.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CF935B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008214C1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DD5490  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001783E59  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000AB8B1D  verint_                   168  verint.F90
oifs_43r3_model.e  0000000000AB7C0B  gpgeo_                    139  gpgeo.F90
oifs_43r3_model.e  0000000000F5A02A  ec_phys_                  458  ec_phys.F90
oifs_43r3_model.e  0000000000F4DBC4  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BD7131  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012C5002  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BACBAE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  0000000000688153  stepo_                    327  stepo.F90
oifs_43r3_model.e  00000000004CD1DC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000004B5D85  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  00000000004B4F52  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  000000000049D638  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  0000000000406FBD  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  0000000000401E7F  MAIN__                     96  master.F90
oifs_43r3_model.e  0000000000401E12  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCC34A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCDBE7  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000401CD5  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a385
(argv4) batchid: 1017
(argv5) wuid: 12286008
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /more/var/lib/boinc-client/slots/16
Project directory is: /more/var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008
..mkdir for temp folder for results failed
Copying: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a9954dfb891619eed0961f1057f59837 to: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_a385_2016092300_20_1017_12286008.zip
Unzipping the namelist zip file: /more/var/lib/boinc-client/slots/16/oifs_43r3_bl_a385_2016092300_20_1017_12286008.zip
ic_ancil_file: ic_ancil_12286008
ifsdata_file: ifsdata_12286008
climate_data_file: clim_data_12286008
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /more/var/lib/boinc-client/slots/16/ic_ancil_12286008.zip
Unzipping the IC ancils zip file: /more/var/lib/boinc-client/slots/16/ic_ancil_12286008.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_411beeabc3b5df374313efcd0cb5b8ec to: /more/var/lib/boinc-client/slots/16/ifsdata/ifsdata_12286008.zip
Unzipping the ifsdata_zip file: /more/var/lib/boinc-client/slots/16/ifsdata/ifsdata_12286008.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /more/var/lib/boinc-client/slots/16/159l_2/clim_data_12286008.zip
Unzipping the climate data zip file: /more/var/lib/boinc-client/slots/16/159l_2/clim_data_12286008.zip
Checking for progress XML file: /more/var/lib/boinc-client/slots/16/progress_file_12286008.xml
Opened progress file ok : /more/var/lib/boinc-client/slots/16/progress_file_12286008.xml
-- Model is restarting --
Adjusting last_iter, 1405, to previous model restart step.
Creating progress file: /more/var/lib/boinc-client/slots/16/progress_file_12286008.xml
last_cpu_time: 23949
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772
The child process has been launched with process id: 3274
Executing the command: /more/var/lib/boinc-client/slots/16/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3274:3274] [20240608:211353:1717895633:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:15:10 STEP 1392 H= 348:00 +CPU= 16.692
  21:15:26 STEP 1393 H= 348:15 +CPU= 15.642
  21:15:43 STEP 1394 H= 348:30 +CPU= 16.245
  21:15:59 STEP 1395 H= 348:45 +CPU= 15.696
  21:16:16 STEP 1396 H= 349:00 +CPU= 16.242
  21:16:33 STEP 1397 H= 349:15 +CPU= 15.965
  21:16:49 STEP 1398 H= 349:30 +CPU= 15.725
  21:17:07 STEP 1399 H= 349:45 +CPU= 16.580
  21:17:24 STEP 1400 H= 350:00 +CPU= 16.713
  21:17:42 STEP 1401 H= 350:15 +CPU= 16.299
  21:17:58 STEP 1402 H= 350:30 +CPU= 16.283
  21:18:16 STEP 1403 H= 350:45 +CPU= 16.522
  21:18:38 STEP 1404 H= 351:00 +CPU= 21.128
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001404
  21:20:05 STEP 1405 H= 351:15 +CPU= 16.256
  21:20:22 STEP 1406 H= 351:30 +CPU= 16.665
  21:20:39 STEP 1407 H= 351:45 +CPU= 16.173
  21:20:56 STEP 1408 H= 352:00 +CPU= 16.329
  21:21:13 STEP 1409 H= 352:15 +CPU= 16.266
  21:21:29 STEP 1410 H= 352:30 +CPU= 15.748
  21:21:46 STEP 1411 H= 352:45 +CPU= 16.125
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:23:05 STEP 1412 H= 353:00 +CPU= 16.371
  21:23:22 STEP 1413 H= 353:15 +CPU= 16.223
  21:23:38 STEP 1414 H= 353:30 +CPU= 16.184
  21:23:53 STEP 1415 H= 353:45 +CPU= 15.648
  21:24:14 STEP 1416 H= 354:00 +CPU= 20.023
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001416
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001416
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001416
  21:24:30 STEP 1417 H= 354:15 +CPU= 16.546
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:25:48 STEP 1418 H= 354:30 +CPU= 15.747
  21:26:05 STEP 1419 H= 354:45 +CPU= 16.362
  21:26:23 STEP 1420 H= 355:00 +CPU= 16.860
  21:26:40 STEP 1421 H= 355:15 +CPU= 16.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:28:00 STEP 1422 H= 355:30 +CPU= 16.206
  21:28:17 STEP 1423 H= 355:45 +CPU= 16.283
  21:28:35 STEP 1424 H= 356:00 +CPU= 16.434
  21:28:51 STEP 1425 H= 356:15 +CPU= 15.750
  21:29:08 STEP 1426 H= 356:30 +CPU= 15.724
  21:29:25 STEP 1427 H= 356:45 +CPU= 16.085
  21:29:47 STEP 1428 H= 357:00 +CPU= 20.924
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001428
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001428
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001428
  21:30:03 STEP 1429 H= 357:15 +CPU= 16.231
  21:30:20 STEP 1430 H= 357:30 +CPU= 16.229
  21:30:37 STEP 1431 H= 357:45 +CPU= 15.578
  21:30:54 STEP 1432 H= 358:00 +CPU= 16.804
  21:31:11 STEP 1433 H= 358:15 +CPU= 16.266
  21:31:28 STEP 1434 H= 358:30 +CPU= 16.211
  21:31:45 STEP 1435 H= 358:45 +CPU= 16.227
  21:32:03 STEP 1436 H= 359:00 +CPU= 17.388
  21:32:21 STEP 1437 H= 359:15 +CPU= 16.689
  21:32:38 STEP 1438 H= 359:30 +CPU= 16.557
  21:32:56 STEP 1439 H= 359:45 +CPU= 16.853
  21:33:21 STEP 1440 H= 360:00 +CPU= 23.277
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.4290926968E-21      I       0.3885285818E-07      I
ICloud fractionI      0.0000000000E+00      I       0.5066372579E-03      I
I  Relat. Hum. I      0.4798464462E+00      I       0.4864036284E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.8043437429E+01      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    336093604 BYTES IN      7 RECORDS TRANSFERRED IN    0.0007 SECONDS480133.7 Mbytes/s, TOTAL TIME=    0.0082( 8.5%)
 21:33:21 STEP 1440 H= 360:00 +CPU= 23.277
 END CNT3
 NSTEP =  1440 CNT0   000000000  16.719  16.719  17.759   0.941               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         1430/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     28    0       46      1044068     1430 ************    59.9    58.0     1.9    58.2     3.5
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 1167807.6 1167807.6    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 1108794.8 1108794.8     94.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                   53    3729.0    3729.0     16.92      0.00
   9 SPCM     - SPECTRAL COMP.                      48     485.5     485.5      2.00      0.00
  10 SCAN2M - PHYSICS                               49   11112.9   11112.9     46.63      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                  4     599.9     599.9      0.21      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  3      25.6      25.6      0.01      0.00
  17 GRIDFPOS IN CNT4                                4      19.4      19.4      0.01      0.00
  27 RESTART FILES - WRITING                         1    2399.5    2399.5      0.21      0.00
  28 RESTART FILES - READING                         1   58692.0   58692.0      5.03      0.00
  29 SU4FPOS IN CNT4                                 4       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                 4    4147.2    4147.2      1.42      0.00
  31 POSDDH IN STEPO                                 1       1.2       1.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                     49    4978.1    4978.1     20.89      0.00
  39 SU0YOMB                                         1     206.6     206.6      0.02      0.00
  51 SCAN2M   - SL COMM. PART 1                     49      13.0      13.0      0.05      0.00
  54 SPCM     - M TO S/S TO M TRANSP.               48     162.6     162.6      0.67      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.               48      30.6      30.6      0.13      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  3       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      108     261.4     261.4      2.42      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       100     168.4     168.4      1.44      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        100      59.0      59.0      0.50      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       108      96.4      96.4      0.89      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      30.6      30.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            108      19.3      19.3      0.18      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            100      11.9      11.9      0.10      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            108      50.7      50.7      0.47      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            100      26.9      26.9      0.23      0.00
 400 GSTATS                                       6667       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                  6149       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     1167.8 CPU TIME     819.8 VECTOR TIME      819.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     1168.8
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMGGh7zg+001440
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMSHh7zg+001440
Moving to projects directory: /more/var/lib/boinc-client/slots/16/ICMUAh7zg+001440
Adding to the zip: /more/var/lib/boinc-client/slots/16/NODE.001_01
Adding to the zip: /more/var/lib/boinc-client/slots/16/ifs.stat
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001440
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001392
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001368
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001368
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001392
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001416
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001416
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001356
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001428
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001380
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001392
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001404
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001368
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001380
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001404
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001428
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001428
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMUAh7zg+001416
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001356
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001380
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001356
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMSHh7zg+001440
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001404
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12286008/ICMGGh7zg+001440
Zipping up the final file: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
21:36:49 (3219): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a385_2016092300_20_1017_12286008_0_r1938819772_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</fil
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Jun 2024 01:35:43 1494452 22437622 oifs_43r3_bl_a385_2016092300_20_1017_12286008_0 1,295,100 24,768 0.0191
08 Jun 2024 16:09:25 1494452 22437622 oifs_43r3_bl_a385_2016092300_20_1017_12286008_0 864,000 15,571 0.0180


©2024 cpdn.org