climateprediction.net home page
Task 22263481

Task 22263481

Name oifs_43r3_ps_0968_1992050100_123_961_12178612_0
Workunit 12178612
Created 22 Dec 2022, 1:53:09 UTC
Sent 10 Jan 2023, 20:48:03 UTC
Report deadline 9 Feb 2023, 20:48:03 UTC
Received 11 Mar 2023, 5:49:33 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1495246
Run time 10 hours 49 min 15 sec
CPU time 10 hours 33 min 15 sec
Validate state Invalid
Credit 8,951.04
Device peak FLOPS 4.45 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,787.64 MB
Peak swap size 4,978.08 MB
Peak disk usage 5,243.91 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
377 H=1377:00 +CPU= 23.832
  17:50:39 STEP 1378 H=1378:00 +CPU= 15.830
  17:50:54 STEP 1379 H=1379:00 +CPU= 15.204
  17:51:21 STEP 1380 H=1380:00 +CPU= 26.565
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001380
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001380
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001380
  17:51:36 STEP 1381 H=1381:00 +CPU= 15.491
  17:51:52 STEP 1382 H=1382:00 +CPU= 16.115
  17:52:16 STEP 1383 H=1383:00 +CPU= 23.840
  17:52:32 STEP 1384 H=1384:00 +CPU= 15.653
  17:52:47 STEP 1385 H=1385:00 +CPU= 15.376
  17:53:12 STEP 1386 H=1386:00 +CPU= 24.828
  17:53:28 STEP 1387 H=1387:00 +CPU= 15.215
  17:53:44 STEP 1388 H=1388:00 +CPU= 15.881
  17:54:08 STEP 1389 H=1389:00 +CPU= 23.708
  17:54:24 STEP 1390 H=1390:00 +CPU= 15.904
  17:54:40 STEP 1391 H=1391:00 +CPU= 16.043
  17:55:08 STEP 1392 H=1392:00 +CPU= 28.184
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001392
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001392
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001392
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001368
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001368
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001368
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001380
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001380
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001380
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_57.zip
Uploading the intermediate file: upload_file_57.zip
  17:55:24 STEP 1393 H=1393:00 +CPU= 15.596
  17:55:40 STEP 1394 H=1394:00 +CPU= 15.798
  17:56:04 STEP 1395 H=1395:00 +CPU= 23.821
  17:56:20 STEP 1396 H=1396:00 +CPU= 15.994
  17:56:36 STEP 1397 H=1397:00 +CPU= 15.453
  17:57:01 STEP 1398 H=1398:00 +CPU= 24.562
  17:57:17 STEP 1399 H=1399:00 +CPU= 16.009
  17:57:33 STEP 1400 H=1400:00 +CPU= 16.286
  17:57:57 STEP 1401 H=1401:00 +CPU= 23.911
  17:58:13 STEP 1402 H=1402:00 +CPU= 15.894
  17:58:29 STEP 1403 H=1403:00 +CPU= 15.452
  17:58:55 STEP 1404 H=1404:00 +CPU= 26.558
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001404
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001404
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001404
  17:59:11 STEP 1405 H=1405:00 +CPU= 15.332
  17:59:27 STEP 1406 H=1406:00 +CPU= 15.867
  17:59:51 STEP 1407 H=1407:00 +CPU= 23.939
  18:00:07 STEP 1408 H=1408:00 +CPU= 16.002
  18:00:22 STEP 1409 H=1409:00 +CPU= 15.246
  18:00:47 STEP 1410 H=1410:00 +CPU= 24.359
  18:01:03 STEP 1411 H=1411:00 +CPU= 15.477
  18:01:18 STEP 1412 H=1412:00 +CPU= 15.783
  18:01:42 STEP 1413 H=1413:00 +CPU= 23.900
  18:01:58 STEP 1414 H=1414:00 +CPU= 15.982
  18:02:14 STEP 1415 H=1415:00 +CPU= 15.838
  18:02:43 STEP 1416 H=1416:00 +CPU= 28.435
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001416
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001416
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001416
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001392
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001392
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001392
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001404
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001404
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001404
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_58.zip
Uploading the intermediate file: upload_file_58.zip
  18:02:59 STEP 1417 H=1417:00 +CPU= 16.492
  18:03:15 STEP 1418 H=1418:00 +CPU= 15.895
  18:03:39 STEP 1419 H=1419:00 +CPU= 23.680
  18:03:55 STEP 1420 H=1420:00 +CPU= 15.682
  18:04:11 STEP 1421 H=1421:00 +CPU= 15.230
  18:04:35 STEP 1422 H=1422:00 +CPU= 24.248
  18:04:51 STEP 1423 H=1423:00 +CPU= 15.343
  18:05:07 STEP 1424 H=1424:00 +CPU= 15.887
  18:05:31 STEP 1425 H=1425:00 +CPU= 23.923
  18:05:47 STEP 1426 H=1426:00 +CPU= 15.707
  18:06:02 STEP 1427 H=1427:00 +CPU= 15.538
  18:06:29 STEP 1428 H=1428:00 +CPU= 26.380
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001428
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001428
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001428
  18:06:44 STEP 1429 H=1429:00 +CPU= 15.385
  18:07:03 STEP 1430 H=1430:00 +CPU= 18.487
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:07:37 STEP 1431 H=1431:00 +CPU= 22.920
  18:07:52 STEP 1432 H=1432:00 +CPU= 15.467
  18:08:08 STEP 1433 H=1433:00 +CPU= 15.299
  18:08:33 STEP 1434 H=1434:00 +CPU= 25.179
  18:08:48 STEP 1435 H=1435:00 +CPU= 15.339
  18:09:05 STEP 1436 H=1436:00 +CPU= 15.980
  18:09:29 STEP 1437 H=1437:00 +CPU= 24.005
  18:09:45 STEP 1438 H=1438:00 +CPU= 15.844
  18:10:01 STEP 1439 H=1439:00 +CPU= 15.872
  18:10:29 STEP 1440 H=1440:00 +CPU= 28.280
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001440
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001440
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001440
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001416
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001416
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001416
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001428
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001428
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001428
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_59.zip
Uploading the intermediate file: upload_file_59.zip
  18:10:45 STEP 1441 H=1441:00 +CPU= 15.487
  18:11:01 STEP 1442 H=1442:00 +CPU= 16.171
  18:11:25 STEP 1443 H=1443:00 +CPU= 23.922
  18:11:41 STEP 1444 H=1444:00 +CPU= 15.843
  18:11:56 STEP 1445 H=1445:00 +CPU= 15.231
  18:12:21 STEP 1446 H=1446:00 +CPU= 24.355
  18:12:36 STEP 1447 H=1447:00 +CPU= 15.362
  18:12:53 STEP 1448 H=1448:00 +CPU= 16.045
  18:13:17 STEP 1449 H=1449:00 +CPU= 23.946
  18:13:33 STEP 1450 H=1450:00 +CPU= 15.826
  18:13:48 STEP 1451 H=1451:00 +CPU= 15.381
  18:14:15 STEP 1452 H=1452:00 +CPU= 26.304
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001452
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001452
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001452
  18:14:30 STEP 1453 H=1453:00 +CPU= 15.581
  18:14:46 STEP 1454 H=1454:00 +CPU= 15.893
  18:15:10 STEP 1455 H=1455:00 +CPU= 23.934
  18:15:26 STEP 1456 H=1456:00 +CPU= 15.767
  18:15:41 STEP 1457 H=1457:00 +CPU= 15.336
  18:16:06 STEP 1458 H=1458:00 +CPU= 24.430
  18:16:21 STEP 1459 H=1459:00 +CPU= 15.237
  18:16:37 STEP 1460 H=1460:00 +CPU= 15.749
  18:17:02 STEP 1461 H=1461:00 +CPU= 24.490
  18:17:18 STEP 1462 H=1462:00 +CPU= 15.804
  18:17:34 STEP 1463 H=1463:00 +CPU= 15.892
  18:18:02 STEP 1464 H=1464:00 +CPU= 28.046
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001464
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001464
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001464
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001440
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001440
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001440
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001452
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001452
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001452
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_60.zip
Uploading the intermediate file: upload_file_60.zip
  18:18:18 STEP 1465 H=1465:00 +CPU= 15.665
  18:18:34 STEP 1466 H=1466:00 +CPU= 15.786
  18:18:58 STEP 1467 H=1467:00 +CPU= 23.837
  18:19:14 STEP 1468 H=1468:00 +CPU= 16.088
  18:19:30 STEP 1469 H=1469:00 +CPU= 15.420
  18:19:54 STEP 1470 H=1470:00 +CPU= 24.416
  18:20:10 STEP 1471 H=1471:00 +CPU= 15.375
  18:20:26 STEP 1472 H=1472:00 +CPU= 16.008
  18:20:50 STEP 1473 H=1473:00 +CPU= 24.220
  18:21:06 STEP 1474 H=1474:00 +CPU= 15.859
  18:21:22 STEP 1475 H=1475:00 +CPU= 15.432
  18:21:48 STEP 1476 H=1476:00 +CPU= 26.265
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001476
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001476
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001476
  18:22:04 STEP 1477 H=1477:00 +CPU= 15.345
  18:22:20 STEP 1478 H=1478:00 +CPU= 15.908
  18:22:44 STEP 1479 H=1479:00 +CPU= 23.771
  18:23:00 STEP 1480 H=1480:00 +CPU= 15.898
  18:23:15 STEP 1481 H=1481:00 +CPU= 15.365
  18:23:40 STEP 1482 H=1482:00 +CPU= 24.490
  18:23:56 STEP 1483 H=1483:00 +CPU= 15.543
  18:24:12 STEP 1484 H=1484:00 +CPU= 16.073
  18:24:36 STEP 1485 H=1485:00 +CPU= 23.787
  18:24:52 STEP 1486 H=1486:00 +CPU= 15.864
  18:25:08 STEP 1487 H=1487:00 +CPU= 15.883
  18:25:36 STEP 1488 H=1488:00 +CPU= 28.242
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001488
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001488
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001488
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001464
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001464
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001464
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001476
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001476
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001476
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_61.zip
Uploading the intermediate file: upload_file_61.zip
  18:25:52 STEP 1489 H=1489:00 +CPU= 15.845
  18:26:08 STEP 1490 H=1490:00 +CPU= 16.105
  18:26:32 STEP 1491 H=1491:00 +CPU= 23.895
  18:26:48 STEP 1492 H=1492:00 +CPU= 15.995
  18:27:04 STEP 1493 H=1493:00 +CPU= 15.321
  18:27:28 STEP 1494 H=1494:00 +CPU= 24.617
  18:27:44 STEP 1495 H=1495:00 +CPU= 15.291
  18:28:00 STEP 1496 H=1496:00 +CPU= 15.893
  18:28:24 STEP 1497 H=1497:00 +CPU= 24.255
  18:28:40 STEP 1498 H=1498:00 +CPU= 15.748
  18:28:56 STEP 1499 H=1499:00 +CPU= 15.355
  18:29:22 STEP 1500 H=1500:00 +CPU= 26.252
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001500
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001500
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001500
  18:29:38 STEP 1501 H=1501:00 +CPU= 15.420
  18:29:54 STEP 1502 H=1502:00 +CPU= 16.099
  18:30:18 STEP 1503 H=1503:00 +CPU= 24.050
  18:30:34 STEP 1504 H=1504:00 +CPU= 15.886
  18:30:50 STEP 1505 H=1505:00 +CPU= 15.360
  18:31:14 STEP 1506 H=1506:00 +CPU= 24.481
  18:31:30 STEP 1507 H=1507:00 +CPU= 15.180
  18:31:46 STEP 1508 H=1508:00 +CPU= 15.788
  18:32:09 STEP 1509 H=1509:00 +CPU= 23.889
  18:32:26 STEP 1510 H=1510:00 +CPU= 16.144
  18:32:42 STEP 1511 H=1511:00 +CPU= 15.960
  18:33:10 STEP 1512 H=1512:00 +CPU= 28.044
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001512
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001512
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001512
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001488
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001488
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001488
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001500
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001500
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001500
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_62.zip
Uploading the intermediate file: upload_file_62.zip
  18:33:26 STEP 1513 H=1513:00 +CPU= 15.815
  18:33:42 STEP 1514 H=1514:00 +CPU= 15.873
  18:34:06 STEP 1515 H=1515:00 +CPU= 23.902
  18:34:22 STEP 1516 H=1516:00 +CPU= 15.838
  18:34:37 STEP 1517 H=1517:00 +CPU= 15.209
  18:35:02 STEP 1518 H=1518:00 +CPU= 24.433
  18:35:17 STEP 1519 H=1519:00 +CPU= 15.269
  18:35:33 STEP 1520 H=1520:00 +CPU= 15.914
  18:35:57 STEP 1521 H=1521:00 +CPU= 23.460
  18:36:13 STEP 1522 H=1522:00 +CPU= 15.941
  18:36:29 STEP 1523 H=1523:00 +CPU= 15.331
  18:36:55 STEP 1524 H=1524:00 +CPU= 26.354
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001524
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001524
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001524
  18:37:11 STEP 1525 H=1525:00 +CPU= 15.627
  18:37:27 STEP 1526 H=1526:00 +CPU= 15.840
  18:37:51 STEP 1527 H=1527:00 +CPU= 24.196
  18:38:07 STEP 1528 H=1528:00 +CPU= 15.941
  18:38:22 STEP 1529 H=1529:00 +CPU= 15.170
  18:38:47 STEP 1530 H=1530:00 +CPU= 24.866
  18:39:03 STEP 1531 H=1531:00 +CPU= 15.301
  18:39:19 STEP 1532 H=1532:00 +CPU= 15.891
  18:39:43 STEP 1533 H=1533:00 +CPU= 23.496
  18:39:58 STEP 1534 H=1534:00 +CPU= 15.781
  18:40:15 STEP 1535 H=1535:00 +CPU= 16.022
  18:40:43 STEP 1536 H=1536:00 +CPU= 27.832
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001536
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001536
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001536
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001512
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001512
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001512
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001524
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001524
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001524
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_63.zip
Uploading the intermediate file: upload_file_63.zip
  18:40:59 STEP 1537 H=1537:00 +CPU= 15.875
Uploading trickle at timestep: 5529600
  18:41:15 STEP 1538 H=1538:00 +CPU= 16.022
  18:41:39 STEP 1539 H=1539:00 +CPU= 23.873
  18:41:55 STEP 1540 H=1540:00 +CPU= 16.077
  18:42:11 STEP 1541 H=1541:00 +CPU= 15.436
  18:42:35 STEP 1542 H=1542:00 +CPU= 24.416
  18:42:51 STEP 1543 H=1543:00 +CPU= 15.440
  18:43:07 STEP 1544 H=1544:00 +CPU= 16.007
  18:43:31 STEP 1545 H=1545:00 +CPU= 24.388
  18:43:47 STEP 1546 H=1546:00 +CPU= 15.937
  18:44:03 STEP 1547 H=1547:00 +CPU= 15.142
  18:44:29 STEP 1548 H=1548:00 +CPU= 26.668
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001548
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001548
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001548
  18:44:45 STEP 1549 H=1549:00 +CPU= 15.360
  18:45:01 STEP 1550 H=1550:00 +CPU= 15.665
  18:45:25 STEP 1551 H=1551:00 +CPU= 23.774
  18:45:41 STEP 1552 H=1552:00 +CPU= 15.679
  18:45:56 STEP 1553 H=1553:00 +CPU= 15.549
  18:46:21 STEP 1554 H=1554:00 +CPU= 24.131
  18:46:36 STEP 1555 H=1555:00 +CPU= 15.186
  18:46:52 STEP 1556 H=1556:00 +CPU= 16.031
  18:47:16 STEP 1557 H=1557:00 +CPU= 24.253
  18:47:32 STEP 1558 H=1558:00 +CPU= 15.858
  18:47:48 STEP 1559 H=1559:00 +CPU= 15.740
  18:48:17 STEP 1560 H=1560:00 +CPU= 28.333
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001560
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001560
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001560
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001536
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001536
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001536
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001548
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001548
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001548
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_64.zip
Uploading the intermediate file: upload_file_64.zip
  18:48:32 STEP 1561 H=1561:00 +CPU= 15.611
  18:48:48 STEP 1562 H=1562:00 +CPU= 15.859
  18:49:12 STEP 1563 H=1563:00 +CPU= 23.630
  18:49:28 STEP 1564 H=1564:00 +CPU= 15.613
  18:49:44 STEP 1565 H=1565:00 +CPU= 15.567
  18:50:08 STEP 1566 H=1566:00 +CPU= 24.097
  18:50:23 STEP 1567 H=1567:00 +CPU= 15.150
  18:50:39 STEP 1568 H=1568:00 +CPU= 15.786
  18:51:03 STEP 1569 H=1569:00 +CPU= 24.004
  18:51:19 STEP 1570 H=1570:00 +CPU= 15.916
  18:51:35 STEP 1571 H=1571:00 +CPU= 15.379
  18:52:01 STEP 1572 H=1572:00 +CPU= 26.421
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001572
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001572
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001572
  18:52:17 STEP 1573 H=1573:00 +CPU= 15.425
  18:52:33 STEP 1574 H=1574:00 +CPU= 15.872
  18:52:57 STEP 1575 H=1575:00 +CPU= 23.691
  18:53:13 STEP 1576 H=1576:00 +CPU= 15.882
  18:53:28 STEP 1577 H=1577:00 +CPU= 15.241
  18:53:53 STEP 1578 H=1578:00 +CPU= 24.335
  18:54:08 STEP 1579 H=1579:00 +CPU= 15.135
  18:54:24 STEP 1580 H=1580:00 +CPU= 15.599
  18:54:48 STEP 1581 H=1581:00 +CPU= 24.042
  18:55:04 STEP 1582 H=1582:00 +CPU= 15.813
  18:55:20 STEP 1583 H=1583:00 +CPU= 15.690
  18:55:48 STEP 1584 H=1584:00 +CPU= 28.022
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001584
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001584
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001584
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001560
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001560
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001560
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001572
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001572
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001572
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_65.zip
Uploading the intermediate file: upload_file_65.zip
  18:56:04 STEP 1585 H=1585:00 +CPU= 15.406
  18:56:20 STEP 1586 H=1586:00 +CPU= 16.024
  18:56:44 STEP 1587 H=1587:00 +CPU= 24.015
  18:57:00 STEP 1588 H=1588:00 +CPU= 15.963
  18:57:15 STEP 1589 H=1589:00 +CPU= 15.369
  18:57:40 STEP 1590 H=1590:00 +CPU= 24.434
  18:57:55 STEP 1591 H=1591:00 +CPU= 15.435
  18:58:12 STEP 1592 H=1592:00 +CPU= 15.873
  18:58:35 STEP 1593 H=1593:00 +CPU= 23.742
  18:58:51 STEP 1594 H=1594:00 +CPU= 15.777
  18:59:07 STEP 1595 H=1595:00 +CPU= 15.296
  18:59:33 STEP 1596 H=1596:00 +CPU= 26.255
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001596
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001596
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001596
  18:59:49 STEP 1597 H=1597:00 +CPU= 15.169
  19:00:05 STEP 1598 H=1598:00 +CPU= 16.054
  19:00:28 STEP 1599 H=1599:00 +CPU= 23.660
  19:00:44 STEP 1600 H=1600:00 +CPU= 15.993
  19:01:00 STEP 1601 H=1601:00 +CPU= 15.382
  19:01:25 STEP 1602 H=1602:00 +CPU= 24.511
  19:01:40 STEP 1603 H=1603:00 +CPU= 15.278
  19:01:56 STEP 1604 H=1604:00 +CPU= 15.929
  19:02:20 STEP 1605 H=1605:00 +CPU= 23.611
  19:02:36 STEP 1606 H=1606:00 +CPU= 15.850
  19:02:52 STEP 1607 H=1607:00 +CPU= 15.789
  19:03:20 STEP 1608 H=1608:00 +CPU= 27.909
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001608
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001608
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001608
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001584
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001584
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001584
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001596
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001596
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001596
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_66.zip
Uploading the intermediate file: upload_file_66.zip
  19:03:36 STEP 1609 H=1609:00 +CPU= 15.878
  19:03:52 STEP 1610 H=1610:00 +CPU= 16.126
  19:04:16 STEP 1611 H=1611:00 +CPU= 23.927
  19:04:32 STEP 1612 H=1612:00 +CPU= 15.907
  19:04:48 STEP 1613 H=1613:00 +CPU= 15.409
  19:05:12 STEP 1614 H=1614:00 +CPU= 24.596
  19:05:28 STEP 1615 H=1615:00 +CPU= 15.464
  19:05:44 STEP 1616 H=1616:00 +CPU= 15.856
  19:06:08 STEP 1617 H=1617:00 +CPU= 23.971
  19:06:24 STEP 1618 H=1618:00 +CPU= 15.764
  19:06:39 STEP 1619 H=1619:00 +CPU= 15.371
  19:07:06 STEP 1620 H=1620:00 +CPU= 26.649
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001620
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001620
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001620
  19:07:22 STEP 1621 H=1621:00 +CPU= 15.518
  19:07:38 STEP 1622 H=1622:00 +CPU= 16.191
  19:08:02 STEP 1623 H=1623:00 +CPU= 23.595
  19:08:18 STEP 1624 H=1624:00 +CPU= 15.825
  19:08:34 STEP 1625 H=1625:00 +CPU= 16.154
  19:08:59 STEP 1626 H=1626:00 +CPU= 24.204
  19:09:14 STEP 1627 H=1627:00 +CPU= 15.300
  19:09:30 STEP 1628 H=1628:00 +CPU= 15.788
  19:09:54 STEP 1629 H=1629:00 +CPU= 24.063
  19:10:10 STEP 1630 H=1630:00 +CPU= 16.013
  19:10:26 STEP 1631 H=1631:00 +CPU= 15.887
  19:10:55 STEP 1632 H=1632:00 +CPU= 28.412
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001632
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001632
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001632
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001608
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001608
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001608
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001620
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001620
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001620
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_67.zip
Uploading the intermediate file: upload_file_67.zip
  19:11:11 STEP 1633 H=1633:00 +CPU= 16.089
  19:11:27 STEP 1634 H=1634:00 +CPU= 16.184
  19:11:52 STEP 1635 H=1635:00 +CPU= 23.926
  19:12:07 STEP 1636 H=1636:00 +CPU= 15.876
  19:12:23 STEP 1637 H=1637:00 +CPU= 15.582
  19:12:48 STEP 1638 H=1638:00 +CPU= 24.323
  20:11:45 STEP 1639 H=1639:00 +CPU= 15.747
  20:12:05 STEP 1640 H=1640:00 +CPU= 17.908
  20:12:32 STEP 1641 H=1641:00 +CPU= 26.274
  20:12:52 STEP 1642 H=1642:00 +CPU= 18.052
  20:13:09 STEP 1643 H=1643:00 +CPU= 16.378
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:13:50 STEP 1644 H=1644:00 +CPU= 28.234
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001644
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001644
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001644
  20:14:07 STEP 1645 H=1645:00 +CPU= 16.246
  20:14:24 STEP 1646 H=1646:00 +CPU= 16.508
  20:14:50 STEP 1647 H=1647:00 +CPU= 24.859
  20:15:09 STEP 1648 H=1648:00 +CPU= 17.445
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:15:50 STEP 1649 H=1649:00 +CPU= 17.751
  20:16:14 STEP 1650 H=1650:00 +CPU= 23.547
  20:16:31 STEP 1651 H=1651:00 +CPU= 16.714
  20:16:49 STEP 1652 H=1652:00 +CPU= 17.027
  20:17:17 STEP 1653 H=1653:00 +CPU= 26.576
  20:17:34 STEP 1654 H=1654:00 +CPU= 16.473
  20:17:51 STEP 1655 H=1655:00 +CPU= 16.530
  20:18:20 STEP 1656 H=1656:00 +CPU= 28.811
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001656
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001656
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001656
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001632
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001632
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001632
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001644
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001644
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001644
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_68.zip
Uploading the intermediate file: upload_file_68.zip
  20:18:37 STEP 1657 H=1657:00 +CPU= 16.453
  20:18:54 STEP 1658 H=1658:00 +CPU= 16.552
  20:19:20 STEP 1659 H=1659:00 +CPU= 24.851
  20:19:36 STEP 1660 H=1660:00 +CPU= 16.470
  20:19:53 STEP 1661 H=1661:00 +CPU= 16.080
  20:20:19 STEP 1662 H=1662:00 +CPU= 25.327
  20:20:35 STEP 1663 H=1663:00 +CPU= 15.905
  20:20:51 STEP 1664 H=1664:00 +CPU= 16.372
  20:21:17 STEP 1665 H=1665:00 +CPU= 24.789
  20:21:33 STEP 1666 H=1666:00 +CPU= 16.462
  20:21:50 STEP 1667 H=1667:00 +CPU= 16.059
  20:22:18 STEP 1668 H=1668:00 +CPU= 27.477
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001668
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001668
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001668
  20:22:34 STEP 1669 H=1669:00 +CPU= 15.925
  20:22:51 STEP 1670 H=1670:00 +CPU= 16.448
  20:23:16 STEP 1671 H=1671:00 +CPU= 24.823
  20:23:33 STEP 1672 H=1672:00 +CPU= 16.388
  20:23:49 STEP 1673 H=1673:00 +CPU= 15.919
  20:24:15 STEP 1674 H=1674:00 +CPU= 25.297
  20:24:31 STEP 1675 H=1675:00 +CPU= 15.943
  20:24:48 STEP 1676 H=1676:00 +CPU= 16.503
  20:25:13 STEP 1677 H=1677:00 +CPU= 24.901
  20:25:30 STEP 1678 H=1678:00 +CPU= 16.563
  20:25:47 STEP 1679 H=1679:00 +CPU= 16.506
  20:26:17 STEP 1680 H=1680:00 +CPU= 29.424
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001680
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001680
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001680
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001656
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001656
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001656
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001668
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001668
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001668
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_69.zip
Uploading the intermediate file: upload_file_69.zip
  20:26:33 STEP 1681 H=1681:00 +CPU= 16.207
  20:26:50 STEP 1682 H=1682:00 +CPU= 16.796
  20:27:16 STEP 1683 H=1683:00 +CPU= 25.097
  20:27:33 STEP 1684 H=1684:00 +CPU= 16.609
  20:27:49 STEP 1685 H=1685:00 +CPU= 15.979
  20:28:15 STEP 1686 H=1686:00 +CPU= 25.484
  20:28:32 STEP 1687 H=1687:00 +CPU= 16.042
  20:28:48 STEP 1688 H=1688:00 +CPU= 16.514
  20:29:14 STEP 1689 H=1689:00 +CPU= 24.974
  20:29:31 STEP 1690 H=1690:00 +CPU= 16.731
  20:29:47 STEP 1691 H=1691:00 +CPU= 16.212
  20:30:15 STEP 1692 H=1692:00 +CPU= 27.604
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001692
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001692
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001692
  20:30:32 STEP 1693 H=1693:00 +CPU= 16.071
  20:30:49 STEP 1694 H=1694:00 +CPU= 16.597
  20:31:14 STEP 1695 H=1695:00 +CPU= 24.970
  20:31:31 STEP 1696 H=1696:00 +CPU= 16.576
  20:31:48 STEP 1697 H=1697:00 +CPU= 16.337
  20:32:14 STEP 1698 H=1698:00 +CPU= 25.500
  20:32:30 STEP 1699 H=1699:00 +CPU= 16.106
  20:32:47 STEP 1700 H=1700:00 +CPU= 16.598
  20:33:12 STEP 1701 H=1701:00 +CPU= 25.001
  20:33:29 STEP 1702 H=1702:00 +CPU= 16.569
  20:33:46 STEP 1703 H=1703:00 +CPU= 16.641
  20:34:16 STEP 1704 H=1704:00 +CPU= 29.337
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001704
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001704
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001704
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001680
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001680
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001680
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001692
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001692
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001692
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_70.zip
Uploading the intermediate file: upload_file_70.zip
  20:34:33 STEP 1705 H=1705:00 +CPU= 16.538
  20:34:50 STEP 1706 H=1706:00 +CPU= 16.649
  20:35:16 STEP 1707 H=1707:00 +CPU= 25.138
  20:35:33 STEP 1708 H=1708:00 +CPU= 16.663
  20:35:49 STEP 1709 H=1709:00 +CPU= 16.115
  20:36:15 STEP 1710 H=1710:00 +CPU= 25.558
  20:36:32 STEP 1711 H=1711:00 +CPU= 16.144
  20:36:48 STEP 1712 H=1712:00 +CPU= 16.601
  20:37:14 STEP 1713 H=1713:00 +CPU= 25.374
  20:37:31 STEP 1714 H=1714:00 +CPU= 16.687
  20:37:48 STEP 1715 H=1715:00 +CPU= 16.078
  20:38:16 STEP 1716 H=1716:00 +CPU= 27.699
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001716
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001716
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001716
  20:38:32 STEP 1717 H=1717:00 +CPU= 16.122
  20:38:49 STEP 1718 H=1718:00 +CPU= 16.586
  20:39:15 STEP 1719 H=1719:00 +CPU= 24.965
  20:39:32 STEP 1720 H=1720:00 +CPU= 16.621
  20:39:48 STEP 1721 H=1721:00 +CPU= 16.148
  20:40:14 STEP 1722 H=1722:00 +CPU= 25.510
  20:40:30 STEP 1723 H=1723:00 +CPU= 16.083
  20:40:47 STEP 1724 H=1724:00 +CPU= 16.509
  20:41:13 STEP 1725 H=1725:00 +CPU= 24.868
  20:41:29 STEP 1726 H=1726:00 +CPU= 16.615
  20:41:46 STEP 1727 H=1727:00 +CPU= 16.660
  20:42:17 STEP 1728 H=1728:00 +CPU= 29.470
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001728
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001728
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001728
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001704
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001704
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001704
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001716
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001716
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001716
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_71.zip
Uploading the intermediate file: upload_file_71.zip
  20:42:33 STEP 1729 H=1729:00 +CPU= 16.506
  20:42:50 STEP 1730 H=1730:00 +CPU= 16.714
  20:43:16 STEP 1731 H=1731:00 +CPU= 25.017
  20:43:33 STEP 1732 H=1732:00 +CPU= 16.595
  20:43:49 STEP 1733 H=1733:00 +CPU= 16.148
  20:44:15 STEP 1734 H=1734:00 +CPU= 25.482
  20:44:32 STEP 1735 H=1735:00 +CPU= 16.018
  20:44:48 STEP 1736 H=1736:00 +CPU= 16.567
  20:45:14 STEP 1737 H=1737:00 +CPU= 25.013
  20:45:31 STEP 1738 H=1738:00 +CPU= 16.600
  20:45:48 STEP 1739 H=1739:00 +CPU= 16.124
  20:46:17 STEP 1740 H=1740:00 +CPU= 27.854
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001740
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001740
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001740
  20:46:33 STEP 1741 H=1741:00 +CPU= 16.243
  20:46:50 STEP 1742 H=1742:00 +CPU= 16.747
  20:47:16 STEP 1743 H=1743:00 +CPU= 25.325
  20:47:35 STEP 1744 H=1744:00 +CPU= 18.170
  20:47:51 STEP 1745 H=1745:00 +CPU= 15.984
  20:48:16 STEP 1746 H=1746:00 +CPU= 25.075
  20:48:32 STEP 1747 H=1747:00 +CPU= 15.754
  20:48:49 STEP 1748 H=1748:00 +CPU= 16.372
  20:49:14 STEP 1749 H=1749:00 +CPU= 24.512
  20:49:30 STEP 1750 H=1750:00 +CPU= 16.344
  20:49:47 STEP 1751 H=1751:00 +CPU= 16.415
  20:50:16 STEP 1752 H=1752:00 +CPU= 28.576
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001752
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001752
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001752
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001728
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001728
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001728
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001740
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001740
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001740
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_72.zip
Uploading the intermediate file: upload_file_72.zip
  20:50:32 STEP 1753 H=1753:00 +CPU= 16.046
  20:50:49 STEP 1754 H=1754:00 +CPU= 16.555
  20:51:14 STEP 1755 H=1755:00 +CPU= 24.840
  20:51:31 STEP 1756 H=1756:00 +CPU= 16.330
  20:51:47 STEP 1757 H=1757:00 +CPU= 15.944
  20:52:13 STEP 1758 H=1758:00 +CPU= 25.288
  20:52:29 STEP 1759 H=1759:00 +CPU= 15.852
  20:52:46 STEP 1760 H=1760:00 +CPU= 16.361
  20:53:10 STEP 1761 H=1761:00 +CPU= 24.429
  20:53:27 STEP 1762 H=1762:00 +CPU= 16.352
  20:53:43 STEP 1763 H=1763:00 +CPU= 15.876
  20:54:11 STEP 1764 H=1764:00 +CPU= 27.263
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001764
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001764
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001764
  20:54:27 STEP 1765 H=1765:00 +CPU= 15.906
  20:54:43 STEP 1766 H=1766:00 +CPU= 16.415
  20:55:08 STEP 1767 H=1767:00 +CPU= 24.578
  20:55:25 STEP 1768 H=1768:00 +CPU= 16.542
  20:55:41 STEP 1769 H=1769:00 +CPU= 15.939
  20:56:07 STEP 1770 H=1770:00 +CPU= 25.326
  20:56:23 STEP 1771 H=1771:00 +CPU= 16.010
  20:56:40 STEP 1772 H=1772:00 +CPU= 16.453
  20:57:05 STEP 1773 H=1773:00 +CPU= 24.519
  20:57:21 STEP 1774 H=1774:00 +CPU= 16.440
  20:57:38 STEP 1775 H=1775:00 +CPU= 16.525
  20:58:07 STEP 1776 H=1776:00 +CPU= 28.679
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001776
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001776
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001776
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001752
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001752
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001752
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001764
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001764
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001764
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_73.zip
Uploading the intermediate file: upload_file_73.zip
  20:58:23 STEP 1777 H=1777:00 +CPU= 15.872
Uploading trickle at timestep: 6393600
  20:58:40 STEP 1778 H=1778:00 +CPU= 16.525
  20:59:05 STEP 1779 H=1779:00 +CPU= 24.689
  20:59:22 STEP 1780 H=1780:00 +CPU= 16.255
  20:59:38 STEP 1781 H=1781:00 +CPU= 15.900
  21:00:03 STEP 1782 H=1782:00 +CPU= 25.221
  21:00:19 STEP 1783 H=1783:00 +CPU= 15.837
  21:00:36 STEP 1784 H=1784:00 +CPU= 16.463
  21:01:01 STEP 1785 H=1785:00 +CPU= 24.657
  21:01:18 STEP 1786 H=1786:00 +CPU= 16.337
  21:01:34 STEP 1787 H=1787:00 +CPU= 15.871
  21:02:01 STEP 1788 H=1788:00 +CPU= 27.270
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001788
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001788
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001788
  21:02:17 STEP 1789 H=1789:00 +CPU= 15.867
  21:02:34 STEP 1790 H=1790:00 +CPU= 16.417
  21:02:59 STEP 1791 H=1791:00 +CPU= 24.623
  21:03:16 STEP 1792 H=1792:00 +CPU= 16.399
  21:03:32 STEP 1793 H=1793:00 +CPU= 15.784
  21:03:57 STEP 1794 H=1794:00 +CPU= 25.113
  21:04:13 STEP 1795 H=1795:00 +CPU= 15.773
  21:04:30 STEP 1796 H=1796:00 +CPU= 16.343
  21:04:55 STEP 1797 H=1797:00 +CPU= 24.640
  21:05:11 STEP 1798 H=1798:00 +CPU= 16.400
  21:05:28 STEP 1799 H=1799:00 +CPU= 16.332
  21:05:57 STEP 1800 H=1800:00 +CPU= 28.831
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001800
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001800
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001800
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001776
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001776
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001776
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001788
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001788
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001788
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_74.zip
Uploading the intermediate file: upload_file_74.zip
  21:06:14 STEP 1801 H=1801:00 +CPU= 16.291
  21:06:30 STEP 1802 H=1802:00 +CPU= 16.416
  21:06:55 STEP 1803 H=1803:00 +CPU= 24.715
  21:07:12 STEP 1804 H=1804:00 +CPU= 16.777
  21:07:29 STEP 1805 H=1805:00 +CPU= 15.943
  21:07:54 STEP 1806 H=1806:00 +CPU= 25.206
  21:08:10 STEP 1807 H=1807:00 +CPU= 15.875
  21:08:27 STEP 1808 H=1808:00 +CPU= 16.362
  21:08:52 STEP 1809 H=1809:00 +CPU= 24.501
  21:09:08 STEP 1810 H=1810:00 +CPU= 16.381
  21:09:24 STEP 1811 H=1811:00 +CPU= 15.818
  21:09:52 STEP 1812 H=1812:00 +CPU= 27.306
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001812
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001812
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001812
  21:10:08 STEP 1813 H=1813:00 +CPU= 15.819
  21:10:25 STEP 1814 H=1814:00 +CPU= 16.459
  21:10:50 STEP 1815 H=1815:00 +CPU= 24.503
  21:11:06 STEP 1816 H=1816:00 +CPU= 16.500
  21:11:22 STEP 1817 H=1817:00 +CPU= 15.821
  21:11:50 STEP 1818 H=1818:00 +CPU= 26.422
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4749MB (heap), 4788MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4749 MB (maxheap), 4788 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211154:1678479114:18265.869] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211155:1678479115:18266.870] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:2534:2534] [20230310:211155:1678479115:18266.870] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DEB667  larmes_                   455  larmes.F90
oifs_43r3_model.e  0000000000DE50FD  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D7A4EC  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0968
(argv4) batchid: 961
(argv5) wuid: 12178612
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/boinc-client/slots/3
Project directory is: /home/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612
..mkdir for temp folder for results failed
Copying: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_9460076dbe4721ad4e8181affe7ce783 to: /home/boinc-client/slots/3/oifs_43r3_ps_0968_1992050100_123_961_12178612.zip
Unzipping the namelist zip file: /home/boinc-client/slots/3/oifs_43r3_ps_0968_1992050100_123_961_12178612.zip
ic_ancil_file: ic_ancil_12178612
ifsdata_file: ifsdata_12178612
climate_data_file: clim_data_12178612
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_8a850dbec80c38ff80ff507ad29d4a39 to: /home/boinc-client/slots/3/ic_ancil_12178612.zip
Unzipping the IC ancils zip file: /home/boinc-client/slots/3/ic_ancil_12178612.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_6bc1dd0f68b22b2e908ab36a2bd822d4 to: /home/boinc-client/slots/3/ifsdata/ifsdata_12178612.zip
Unzipping the ifsdata_zip file: /home/boinc-client/slots/3/ifsdata/ifsdata_12178612.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/boinc-client/slots/3/159l_2/clim_data_12178612.zip
Unzipping the climate data zip file: /home/boinc-client/slots/3/159l_2/clim_data_12178612.zip
Checking for progress XML file: /home/boinc-client/slots/3/progress_file_12178612.xml
Opened progress file ok : /home/boinc-client/slots/3/progress_file_12178612.xml
-- Model is restarting --
Adjusting last_iter, 1818, to previous model restart step.
Creating progress file: /home/boinc-client/slots/3/progress_file_12178612.xml
last_cpu_time: 37043
upload_file_number: 75
last_iter: 1799
last_upload: 6480000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163
The child process has been launched with process id: 1549
Executing the command: /home/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Hp-ProBook-430-G4:1:1:1549:1549] [20230311:054849:1678510129:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:49:19 STEP 1800 H=1800:00 +CPU= 25.740
  05:49:34 STEP 1801 H=1801:00 +CPU= 15.272
  05:49:50 STEP 1802 H=1802:00 +CPU= 15.866
  05:50:15 STEP 1803 H=1803:00 +CPU= 24.427
  05:50:31 STEP 1804 H=1804:00 +CPU= 15.882
  05:50:46 STEP 1805 H=1805:00 +CPU= 15.236
  05:51:10 STEP 1806 H=1806:00 +CPU= 24.158
  05:51:26 STEP 1807 H=1807:00 +CPU= 15.430
  05:51:41 STEP 1808 H=1808:00 +CPU= 15.511
  05:52:05 STEP 1809 H=1809:00 +CPU= 23.569
  05:52:20 STEP 1810 H=1810:00 +CPU= 15.609
  05:52:36 STEP 1811 H=1811:00 +CPU= 15.542
  05:53:02 STEP 1812 H=1812:00 +CPU= 26.078
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001812
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001812
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001812
  05:53:17 STEP 1813 H=1813:00 +CPU= 15.081
  05:53:33 STEP 1814 H=1814:00 +CPU= 15.578
  05:53:56 STEP 1815 H=1815:00 +CPU= 23.515
  05:54:12 STEP 1816 H=1816:00 +CPU= 15.710
  05:54:27 STEP 1817 H=1817:00 +CPU= 15.165
  05:54:52 STEP 1818 H=1818:00 +CPU= 25.010
  05:55:07 STEP 1819 H=1819:00 +CPU= 15.273
  05:55:23 STEP 1820 H=1820:00 +CPU= 15.717
  05:55:47 STEP 1821 H=1821:00 +CPU= 23.634
  05:56:03 STEP 1822 H=1822:00 +CPU= 15.795
  05:56:18 STEP 1823 H=1823:00 +CPU= 15.564
  05:56:46 STEP 1824 H=1824:00 +CPU= 27.723
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001824
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001824
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001824
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001800
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001800
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001800
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001812
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001812
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001812
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_75.zip
Uploading the intermediate file: upload_file_75.zip
  05:57:01 STEP 1825 H=1825:00 +CPU= 15.470
  05:57:17 STEP 1826 H=1826:00 +CPU= 15.936
  05:57:41 STEP 1827 H=1827:00 +CPU= 23.786
  05:57:57 STEP 1828 H=1828:00 +CPU= 15.651
  05:58:12 STEP 1829 H=1829:00 +CPU= 15.461
  05:58:36 STEP 1830 H=1830:00 +CPU= 24.103
  05:58:52 STEP 1831 H=1831:00 +CPU= 15.079
  05:59:07 STEP 1832 H=1832:00 +CPU= 15.478
  05:59:31 STEP 1833 H=1833:00 +CPU= 23.535
  05:59:46 STEP 1834 H=1834:00 +CPU= 15.609
  06:00:01 STEP 1835 H=1835:00 +CPU= 15.285
  06:00:28 STEP 1836 H=1836:00 +CPU= 26.257
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001836
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001836
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001836
  06:00:43 STEP 1837 H=1837:00 +CPU= 15.345
  06:00:59 STEP 1838 H=1838:00 +CPU= 15.713
  06:01:23 STEP 1839 H=1839:00 +CPU= 23.764
  06:01:38 STEP 1840 H=1840:00 +CPU= 15.663
  06:01:54 STEP 1841 H=1841:00 +CPU= 15.237
  06:02:18 STEP 1842 H=1842:00 +CPU= 24.913
  06:02:34 STEP 1843 H=1843:00 +CPU= 15.409
  06:02:50 STEP 1844 H=1844:00 +CPU= 16.049
  06:03:13 STEP 1845 H=1845:00 +CPU= 23.365
  06:03:29 STEP 1846 H=1846:00 +CPU= 15.556
  06:03:44 STEP 1847 H=1847:00 +CPU= 15.419
  06:04:08 STEP 1848 H=1848:00 +CPU= 23.429
Moving to projects directory: /home/boinc-client/slots/3/ICMGGhq0f+001848
Moving to projects directory: /home/boinc-client/slots/3/ICMSHhq0f+001848
Moving to projects directory: /home/boinc-client/slots/3/ICMUAhq0f+001848
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001824
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001824
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001824
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMGGhq0f+001836
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMSHhq0f+001836
Adding to the zip: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178612/ICMUAhq0f+001836
Zipping up the intermediate file: /home/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0968_1992050100_123_961_12178612_0_r375759163_76.zip
Uploading the intermediate file: upload_file_76.zip
  06:04:21 STEP 1849 H=1849:00 +CPU= 12.912
  06:04:34 STEP 1850 H=1850:00 +CPU= 13.001
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:55:28 1495246 22263481 oifs_43r3_ps_0968_1992050100_123_961_12178612_0 6,393,600 36,241 0.0057
09 May 2023 16:55:28 1495246 22263481 oifs_43r3_ps_0968_1992050100_123_961_12178612_0 5,529,600 31,629 0.0057
09 May 2023 16:55:28 1495246 22263481 oifs_43r3_ps_0968_1992050100_123_961_12178612_0 4,665,600 27,114 0.0058
09 May 2023 16:55:11 1495246 22263481 oifs_43r3_ps_0968_1992050100_123_961_12178612_0 3,628,800 21,401 0.0059
09 May 2023 16:55:04 1495246 22263481 oifs_43r3_ps_0968_1992050100_123_961_12178612_0 2,419,200 14,016 0.0058
09 May 2023 16:55:00 1495246 22263481 oifs_43r3_ps_0968_1992050100_123_961_12178612_0 950,400 5,877 0.0062


©2024 cpdn.org