climateprediction.net home page
Task 22264666

Task 22264666

Name oifs_43r3_ps_0186_1987050100_123_956_12172830_1
Workunit 12172830
Created 22 Dec 2022, 4:51:34 UTC
Sent 1 Jan 2023, 14:06:53 UTC
Report deadline 31 Jan 2023, 14:06:53 UTC
Received 3 Jan 2023, 5:38:54 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1499440
Run time 1 days 10 hours 51 min 19 sec
CPU time 22 hours 26 min 40 sec
Validate state Invalid
Credit 9,192.96
Device peak FLOPS 4.53 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,392.62 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,201.56 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
 the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_61.zip
Uploading the intermediate file: upload_file_61.zip
  21:46:32 STEP 1489 H=1489:00 +CPU= 34.582
  21:47:23 STEP 1490 H=1490:00 +CPU= 35.055
  21:48:17 STEP 1491 H=1491:00 +CPU= 52.934
  21:50:25 STEP 1492 H=1492:00 +CPU= 39.794
  21:52:09 STEP 1493 H=1493:00 +CPU= 36.637
  21:53:23 STEP 1494 H=1494:00 +CPU= 53.952
  21:54:00 STEP 1495 H=1495:00 +CPU= 34.762
  21:54:36 STEP 1496 H=1496:00 +CPU= 35.832
  21:56:32 STEP 1497 H=1497:00 +CPU= 54.282
  21:57:09 STEP 1498 H=1498:00 +CPU= 34.859
  21:57:46 STEP 1499 H=1499:00 +CPU= 35.202
  21:58:48 STEP 1500 H=1500:00 +CPU= 58.070
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001500
  21:59:24 STEP 1501 H=1501:00 +CPU= 33.731
  22:00:14 STEP 1502 H=1502:00 +CPU= 35.579
  22:01:22 STEP 1503 H=1503:00 +CPU= 52.810
  22:01:58 STEP 1504 H=1504:00 +CPU= 35.274
  22:02:35 STEP 1505 H=1505:00 +CPU= 34.604
  22:03:40 STEP 1506 H=1506:00 +CPU= 54.338
  22:04:17 STEP 1507 H=1507:00 +CPU= 34.885
  22:04:53 STEP 1508 H=1508:00 +CPU= 35.354
  22:08:12 STEP 1509 H=1509:00 +CPU= 57.858
  22:08:56 STEP 1510 H=1510:00 +CPU= 35.486
  22:09:33 STEP 1511 H=1511:00 +CPU= 34.935
  22:10:54 STEP 1512 H=1512:00 +CPU= 64.317
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_62.zip
Uploading the intermediate file: upload_file_62.zip
  22:11:43 STEP 1513 H=1513:00 +CPU= 35.063
  22:12:25 STEP 1514 H=1514:00 +CPU= 34.775
  22:13:20 STEP 1515 H=1515:00 +CPU= 52.901
  22:13:57 STEP 1516 H=1516:00 +CPU= 35.801
  22:14:38 STEP 1517 H=1517:00 +CPU= 34.277
  22:15:33 STEP 1518 H=1518:00 +CPU= 53.439
  22:16:10 STEP 1519 H=1519:00 +CPU= 35.096
  22:16:47 STEP 1520 H=1520:00 +CPU= 34.841
  22:17:44 STEP 1521 H=1521:00 +CPU= 55.021
  22:18:23 STEP 1522 H=1522:00 +CPU= 35.419
  22:19:06 STEP 1523 H=1523:00 +CPU= 34.044
  22:23:37 STEP 1524 H=1524:00 +CPU= 62.746
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001524
  22:24:15 STEP 1525 H=1525:00 +CPU= 34.754
  22:24:51 STEP 1526 H=1526:00 +CPU= 35.498
  22:25:47 STEP 1527 H=1527:00 +CPU= 53.494
  22:26:24 STEP 1528 H=1528:00 +CPU= 35.564
  22:26:59 STEP 1529 H=1529:00 +CPU= 34.491
  22:27:55 STEP 1530 H=1530:00 +CPU= 54.250
  22:28:31 STEP 1531 H=1531:00 +CPU= 34.671
  22:29:07 STEP 1532 H=1532:00 +CPU= 35.381
  22:30:03 STEP 1533 H=1533:00 +CPU= 53.022
  22:30:41 STEP 1534 H=1534:00 +CPU= 35.385
  22:31:18 STEP 1535 H=1535:00 +CPU= 35.681
  22:32:42 STEP 1536 H=1536:00 +CPU= 63.781
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_63.zip
Uploading the intermediate file: upload_file_63.zip
  22:33:19 STEP 1537 H=1537:00 +CPU= 34.909
  22:33:55 STEP 1538 H=1538:00 +CPU= 34.687
  22:34:50 STEP 1539 H=1539:00 +CPU= 53.542
  22:35:33 STEP 1540 H=1540:00 +CPU= 35.351
  22:36:18 STEP 1541 H=1541:00 +CPU= 33.962
  22:37:15 STEP 1542 H=1542:00 +CPU= 53.837
  22:37:52 STEP 1543 H=1543:00 +CPU= 34.223
  22:38:28 STEP 1544 H=1544:00 +CPU= 34.750
  22:39:28 STEP 1545 H=1545:00 +CPU= 53.105
  22:40:06 STEP 1546 H=1546:00 +CPU= 35.898
  22:40:41 STEP 1547 H=1547:00 +CPU= 34.257
  22:42:33 STEP 1548 H=1548:00 +CPU= 59.700
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001548
  22:45:15 STEP 1549 H=1549:00 +CPU= 36.402
  22:45:51 STEP 1550 H=1550:00 +CPU= 32.720
  22:47:17 STEP 1551 H=1551:00 +CPU= 49.106
  22:47:51 STEP 1552 H=1552:00 +CPU= 33.435
  22:48:27 STEP 1553 H=1553:00 +CPU= 34.230
  22:49:23 STEP 1554 H=1554:00 +CPU= 54.329
  22:49:59 STEP 1555 H=1555:00 +CPU= 34.584
  22:50:35 STEP 1556 H=1556:00 +CPU= 34.930
  22:51:30 STEP 1557 H=1557:00 +CPU= 53.512
  22:52:06 STEP 1558 H=1558:00 +CPU= 34.952
  22:52:45 STEP 1559 H=1559:00 +CPU= 36.031
  22:54:21 STEP 1560 H=1560:00 +CPU= 64.903
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_64.zip
Uploading the intermediate file: upload_file_64.zip
  22:54:57 STEP 1561 H=1561:00 +CPU= 34.645
  22:55:34 STEP 1562 H=1562:00 +CPU= 35.357
  22:56:29 STEP 1563 H=1563:00 +CPU= 52.988
  22:57:07 STEP 1564 H=1564:00 +CPU= 35.369
  22:57:43 STEP 1565 H=1565:00 +CPU= 34.730
  22:58:38 STEP 1566 H=1566:00 +CPU= 53.622
  22:59:14 STEP 1567 H=1567:00 +CPU= 34.825
  22:59:51 STEP 1568 H=1568:00 +CPU= 35.506
  23:00:45 STEP 1569 H=1569:00 +CPU= 53.171
  23:01:20 STEP 1570 H=1570:00 +CPU= 34.166
  23:02:02 STEP 1571 H=1571:00 +CPU= 34.463
  23:03:11 STEP 1572 H=1572:00 +CPU= 58.440
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001572
  23:03:47 STEP 1573 H=1573:00 +CPU= 34.988
  23:04:27 STEP 1574 H=1574:00 +CPU= 35.100
  23:05:22 STEP 1575 H=1575:00 +CPU= 52.816
  23:05:58 STEP 1576 H=1576:00 +CPU= 35.575
  23:06:34 STEP 1577 H=1577:00 +CPU= 34.225
  23:07:30 STEP 1578 H=1578:00 +CPU= 53.610
  23:08:06 STEP 1579 H=1579:00 +CPU= 34.917
  23:08:42 STEP 1580 H=1580:00 +CPU= 34.885
  23:09:40 STEP 1581 H=1581:00 +CPU= 55.143
  23:10:17 STEP 1582 H=1582:00 +CPU= 35.484
  23:10:57 STEP 1583 H=1583:00 +CPU= 34.980
  23:15:34 STEP 1584 H=1584:00 +CPU= 68.049
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_65.zip
Uploading the intermediate file: upload_file_65.zip
  23:16:33 STEP 1585 H=1585:00 +CPU= 34.524
  23:17:35 STEP 1586 H=1586:00 +CPU= 36.029
  23:19:08 STEP 1587 H=1587:00 +CPU= 52.404
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0186
(argv4) batchid: 956
(argv5) wuid: 12172830
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e40be6eb0a76bc84586cd977698623f1 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0186_1987050100_123_956_12172830.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0186_1987050100_123_956_12172830.zip
ic_ancil_file: ic_ancil_12172830
ifsdata_file: ifsdata_12172830
climate_data_file: clim_data_12172830
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_177aa6b6e839ad18fe54f412db351241 to: /var/lib/boinc-client/slots/5/ic_ancil_12172830.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12172830.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6183187ac7bd66d3acab98b28087d16 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12172830.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12172830.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12172830.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12172830.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12172830.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12172830.xml
-- Model is restarting --
Adjusting last_iter, 1588, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12172830.xml
last_cpu_time: 65095
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328
The child process has been launched with process id: 32578
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:32578:32578] [20230102:232109:1672698069:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:22:34 STEP 1584 H=1584:00 +CPU= 60.372
  23:23:50 STEP 1585 H=1585:00 +CPU= 34.077
  23:25:10 STEP 1586 H=1586:00 +CPU= 37.806
  23:26:12 STEP 1587 H=1587:00 +CPU= 59.259
  23:26:53 STEP 1588 H=1588:00 +CPU= 40.265
  23:27:33 STEP 1589 H=1589:00 +CPU= 39.034
  23:28:36 STEP 1590 H=1590:00 +CPU= 60.271
  23:29:16 STEP 1591 H=1591:00 +CPU= 39.327
  23:29:58 STEP 1592 H=1592:00 +CPU= 40.068
  23:30:59 STEP 1593 H=1593:00 +CPU= 59.309
  23:31:41 STEP 1594 H=1594:00 +CPU= 40.328
  23:32:21 STEP 1595 H=1595:00 +CPU= 39.134
  23:33:29 STEP 1596 H=1596:00 +CPU= 65.074
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001596
  23:34:10 STEP 1597 H=1597:00 +CPU= 39.422
  23:34:51 STEP 1598 H=1598:00 +CPU= 39.619
  23:35:52 STEP 1599 H=1599:00 +CPU= 59.235
  23:36:34 STEP 1600 H=1600:00 +CPU= 40.157
  23:37:14 STEP 1601 H=1601:00 +CPU= 39.069
  23:38:15 STEP 1602 H=1602:00 +CPU= 59.899
  23:38:55 STEP 1603 H=1603:00 +CPU= 38.670
  23:39:39 STEP 1604 H=1604:00 +CPU= 41.934
  23:40:41 STEP 1605 H=1605:00 +CPU= 60.110
  23:41:23 STEP 1606 H=1606:00 +CPU= 40.078
  23:42:05 STEP 1607 H=1607:00 +CPU= 40.438
  23:43:25 STEP 1608 H=1608:00 +CPU= 71.034
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_66.zip
Uploading the intermediate file: upload_file_66.zip
  23:44:05 STEP 1609 H=1609:00 +CPU= 38.568
  23:44:46 STEP 1610 H=1610:00 +CPU= 39.511
  23:45:48 STEP 1611 H=1611:00 +CPU= 59.461
  23:46:29 STEP 1612 H=1612:00 +CPU= 39.707
  23:47:09 STEP 1613 H=1613:00 +CPU= 38.742
  23:48:11 STEP 1614 H=1614:00 +CPU= 60.404
  23:48:51 STEP 1615 H=1615:00 +CPU= 39.078
  23:49:32 STEP 1616 H=1616:00 +CPU= 39.457
  23:50:33 STEP 1617 H=1617:00 +CPU= 59.580
  23:51:14 STEP 1618 H=1618:00 +CPU= 39.631
  23:51:54 STEP 1619 H=1619:00 +CPU= 39.106
  23:53:02 STEP 1620 H=1620:00 +CPU= 65.481
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001620
  23:53:42 STEP 1621 H=1621:00 +CPU= 38.653
  23:54:22 STEP 1622 H=1622:00 +CPU= 39.371
  23:55:24 STEP 1623 H=1623:00 +CPU= 59.412
  23:56:04 STEP 1624 H=1624:00 +CPU= 39.324
  23:56:44 STEP 1625 H=1625:00 +CPU= 38.422
  23:57:46 STEP 1626 H=1626:00 +CPU= 60.468
  23:58:26 STEP 1627 H=1627:00 +CPU= 38.479
  23:59:06 STEP 1628 H=1628:00 +CPU= 39.389
  00:00:08 STEP 1629 H=1629:00 +CPU= 59.409
  00:00:49 STEP 1630 H=1630:00 +CPU= 39.692
  00:01:29 STEP 1631 H=1631:00 +CPU= 39.530
  00:02:43 STEP 1632 H=1632:00 +CPU= 70.764
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_67.zip
Uploading the intermediate file: upload_file_67.zip
  00:03:24 STEP 1633 H=1633:00 +CPU= 39.212
  00:04:05 STEP 1634 H=1634:00 +CPU= 39.652
  00:05:06 STEP 1635 H=1635:00 +CPU= 58.954
  00:05:46 STEP 1636 H=1636:00 +CPU= 39.848
  00:06:26 STEP 1637 H=1637:00 +CPU= 38.840
  00:07:31 STEP 1638 H=1638:00 +CPU= 62.454
  00:08:12 STEP 1639 H=1639:00 +CPU= 40.029
  00:08:53 STEP 1640 H=1640:00 +CPU= 40.176
  00:09:54 STEP 1641 H=1641:00 +CPU= 58.783
  00:10:35 STEP 1642 H=1642:00 +CPU= 39.958
  00:11:16 STEP 1643 H=1643:00 +CPU= 39.245
  00:12:22 STEP 1644 H=1644:00 +CPU= 65.010
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001644
  00:13:02 STEP 1645 H=1645:00 +CPU= 38.491
  00:13:44 STEP 1646 H=1646:00 +CPU= 40.387
  00:15:20 STEP 1647 H=1647:00 +CPU= 54.220
  00:16:05 STEP 1648 H=1648:00 +CPU= 32.981
  00:16:40 STEP 1649 H=1649:00 +CPU= 33.766
  00:17:53 STEP 1650 H=1650:00 +CPU= 53.016
  00:18:26 STEP 1651 H=1651:00 +CPU= 32.098
  00:19:00 STEP 1652 H=1652:00 +CPU= 32.782
  00:19:50 STEP 1653 H=1653:00 +CPU= 49.169
  00:20:23 STEP 1654 H=1654:00 +CPU= 32.663
  00:20:57 STEP 1655 H=1655:00 +CPU= 32.641
  00:22:08 STEP 1656 H=1656:00 +CPU= 58.787
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_68.zip
Uploading the intermediate file: upload_file_68.zip
  00:22:40 STEP 1657 H=1657:00 +CPU= 31.724
  00:23:13 STEP 1658 H=1658:00 +CPU= 32.650
  00:24:56 STEP 1659 H=1659:00 +CPU= 52.425
  00:27:35 STEP 1660 H=1660:00 +CPU= 35.090
  00:28:14 STEP 1661 H=1661:00 +CPU= 34.804
  00:29:11 STEP 1662 H=1662:00 +CPU= 54.552
  00:29:47 STEP 1663 H=1663:00 +CPU= 34.084
  00:30:25 STEP 1664 H=1664:00 +CPU= 35.936
  00:31:20 STEP 1665 H=1665:00 +CPU= 53.522
  00:31:57 STEP 1666 H=1666:00 +CPU= 34.949
  00:32:43 STEP 1667 H=1667:00 +CPU= 34.183
  00:33:46 STEP 1668 H=1668:00 +CPU= 58.926
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001668
  00:34:21 STEP 1669 H=1669:00 +CPU= 34.202
  00:35:00 STEP 1670 H=1670:00 +CPU= 35.932
  00:36:05 STEP 1671 H=1671:00 +CPU= 54.043
  00:36:41 STEP 1672 H=1672:00 +CPU= 34.893
  00:37:18 STEP 1673 H=1673:00 +CPU= 35.103
  00:38:19 STEP 1674 H=1674:00 +CPU= 55.104
  00:38:54 STEP 1675 H=1675:00 +CPU= 33.899
  00:39:32 STEP 1676 H=1676:00 +CPU= 35.983
  00:40:27 STEP 1677 H=1677:00 +CPU= 53.574
  00:41:03 STEP 1678 H=1678:00 +CPU= 34.786
  00:41:42 STEP 1679 H=1679:00 +CPU= 36.288
  00:43:07 STEP 1680 H=1680:00 +CPU= 63.048
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_69.zip
Uploading the intermediate file: upload_file_69.zip
  00:43:50 STEP 1681 H=1681:00 +CPU= 35.450
  00:44:28 STEP 1682 H=1682:00 +CPU= 35.349
  00:45:25 STEP 1683 H=1683:00 +CPU= 52.812
  00:46:02 STEP 1684 H=1684:00 +CPU= 35.574
  00:46:37 STEP 1685 H=1685:00 +CPU= 34.674
  00:47:41 STEP 1686 H=1686:00 +CPU= 56.697
  00:48:21 STEP 1687 H=1687:00 +CPU= 30.374
  00:48:56 STEP 1688 H=1688:00 +CPU= 33.672
  00:49:51 STEP 1689 H=1689:00 +CPU= 53.372
  00:50:30 STEP 1690 H=1690:00 +CPU= 35.375
  00:51:06 STEP 1691 H=1691:00 +CPU= 34.911
  00:52:07 STEP 1692 H=1692:00 +CPU= 58.745
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001692
  00:52:43 STEP 1693 H=1693:00 +CPU= 34.789
  00:53:21 STEP 1694 H=1694:00 +CPU= 35.636
  00:54:17 STEP 1695 H=1695:00 +CPU= 53.007
  00:54:54 STEP 1696 H=1696:00 +CPU= 35.200
  00:55:31 STEP 1697 H=1697:00 +CPU= 34.770
  00:56:27 STEP 1698 H=1698:00 +CPU= 54.188
  00:57:08 STEP 1699 H=1699:00 +CPU= 34.610
  00:57:45 STEP 1700 H=1700:00 +CPU= 35.380
  00:58:40 STEP 1701 H=1701:00 +CPU= 52.946
  00:59:18 STEP 1702 H=1702:00 +CPU= 35.612
  00:59:56 STEP 1703 H=1703:00 +CPU= 35.824
  01:01:08 STEP 1704 H=1704:00 +CPU= 63.445
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_70.zip
Uploading the intermediate file: upload_file_70.zip
  01:04:07 STEP 1705 H=1705:00 +CPU= 36.755
  01:04:45 STEP 1706 H=1706:00 +CPU= 35.337
  01:06:24 STEP 1707 H=1707:00 +CPU= 55.118
  01:07:00 STEP 1708 H=1708:00 +CPU= 34.949
  01:07:36 STEP 1709 H=1709:00 +CPU= 35.325
  01:08:32 STEP 1710 H=1710:00 +CPU= 54.124
  01:09:12 STEP 1711 H=1711:00 +CPU= 34.340
  01:09:49 STEP 1712 H=1712:00 +CPU= 35.425
  01:10:44 STEP 1713 H=1713:00 +CPU= 53.255
  01:11:20 STEP 1714 H=1714:00 +CPU= 34.842
  01:11:58 STEP 1715 H=1715:00 +CPU= 35.283
  01:13:00 STEP 1716 H=1716:00 +CPU= 58.693
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001716
  01:13:35 STEP 1717 H=1717:00 +CPU= 34.267
  01:14:14 STEP 1718 H=1718:00 +CPU= 37.900
  01:15:26 STEP 1719 H=1719:00 +CPU= 53.493
  01:16:03 STEP 1720 H=1720:00 +CPU= 34.987
  01:16:41 STEP 1721 H=1721:00 +CPU= 35.156
  01:17:38 STEP 1722 H=1722:00 +CPU= 54.651
  01:18:13 STEP 1723 H=1723:00 +CPU= 33.971
  01:18:52 STEP 1724 H=1724:00 +CPU= 35.847
  01:19:47 STEP 1725 H=1725:00 +CPU= 52.689
  01:20:23 STEP 1726 H=1726:00 +CPU= 34.706
  01:21:02 STEP 1727 H=1727:00 +CPU= 35.871
  01:22:22 STEP 1728 H=1728:00 +CPU= 64.257
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_71.zip
Uploading the intermediate file: upload_file_71.zip
  01:23:02 STEP 1729 H=1729:00 +CPU= 34.326
  01:23:40 STEP 1730 H=1730:00 +CPU= 35.588
  01:24:35 STEP 1731 H=1731:00 +CPU= 53.185
  01:25:12 STEP 1732 H=1732:00 +CPU= 35.306
  01:25:49 STEP 1733 H=1733:00 +CPU= 34.418
  01:26:46 STEP 1734 H=1734:00 +CPU= 54.626
  01:27:21 STEP 1735 H=1735:00 +CPU= 34.228
  01:27:58 STEP 1736 H=1736:00 +CPU= 35.669
  01:28:53 STEP 1737 H=1737:00 +CPU= 53.292
  01:29:37 STEP 1738 H=1738:00 +CPU= 35.650
  01:30:14 STEP 1739 H=1739:00 +CPU= 34.919
  01:31:18 STEP 1740 H=1740:00 +CPU= 58.308
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001740
  01:31:54 STEP 1741 H=1741:00 +CPU= 34.371
  01:32:34 STEP 1742 H=1742:00 +CPU= 35.349
  01:33:29 STEP 1743 H=1743:00 +CPU= 53.559
  01:34:05 STEP 1744 H=1744:00 +CPU= 35.139
  01:34:42 STEP 1745 H=1745:00 +CPU= 34.797
  01:35:38 STEP 1746 H=1746:00 +CPU= 54.271
  01:36:18 STEP 1747 H=1747:00 +CPU= 34.239
  01:36:55 STEP 1748 H=1748:00 +CPU= 35.395
  01:37:49 STEP 1749 H=1749:00 +CPU= 52.733
  01:38:26 STEP 1750 H=1750:00 +CPU= 35.031
  01:39:05 STEP 1751 H=1751:00 +CPU= 35.608
  01:40:23 STEP 1752 H=1752:00 +CPU= 65.745
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_72.zip
Uploading the intermediate file: upload_file_72.zip
  01:41:01 STEP 1753 H=1753:00 +CPU= 34.482
  01:41:43 STEP 1754 H=1754:00 +CPU= 35.480
  01:42:38 STEP 1755 H=1755:00 +CPU= 52.890
  01:43:15 STEP 1756 H=1756:00 +CPU= 35.529
  01:43:51 STEP 1757 H=1757:00 +CPU= 34.730
  01:44:48 STEP 1758 H=1758:00 +CPU= 53.866
  01:45:25 STEP 1759 H=1759:00 +CPU= 34.793
  01:46:01 STEP 1760 H=1760:00 +CPU= 35.445
  01:46:56 STEP 1761 H=1761:00 +CPU= 52.737
  01:47:34 STEP 1762 H=1762:00 +CPU= 35.488
  01:48:11 STEP 1763 H=1763:00 +CPU= 34.918
  01:49:21 STEP 1764 H=1764:00 +CPU= 58.763
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001764
  01:49:57 STEP 1765 H=1765:00 +CPU= 34.543
  01:50:34 STEP 1766 H=1766:00 +CPU= 35.608
  01:51:41 STEP 1767 H=1767:00 +CPU= 57.494
  01:53:06 STEP 1768 H=1768:00 +CPU= 33.839
  01:53:44 STEP 1769 H=1769:00 +CPU= 34.158
  01:55:30 STEP 1770 H=1770:00 +CPU= 56.112
  01:56:51 STEP 1771 H=1771:00 +CPU= 34.027
  01:57:26 STEP 1772 H=1772:00 +CPU= 33.658
  01:58:26 STEP 1773 H=1773:00 +CPU= 52.228
  01:59:02 STEP 1774 H=1774:00 +CPU= 34.882
  01:59:39 STEP 1775 H=1775:00 +CPU= 34.837
  02:01:28 STEP 1776 H=1776:00 +CPU= 65.404
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_73.zip
Uploading the intermediate file: upload_file_73.zip
  02:02:06 STEP 1777 H=1777:00 +CPU= 34.138
  02:02:43 STEP 1778 H=1778:00 +CPU= 35.702
  02:03:40 STEP 1779 H=1779:00 +CPU= 53.526
  02:04:16 STEP 1780 H=1780:00 +CPU= 34.813
  02:04:52 STEP 1781 H=1781:00 +CPU= 34.877
  02:05:49 STEP 1782 H=1782:00 +CPU= 54.952
  02:06:26 STEP 1783 H=1783:00 +CPU= 35.912
  02:07:03 STEP 1784 H=1784:00 +CPU= 35.591
  02:07:59 STEP 1785 H=1785:00 +CPU= 53.544
  02:08:34 STEP 1786 H=1786:00 +CPU= 35.266
  02:09:11 STEP 1787 H=1787:00 +CPU= 34.765
  02:10:13 STEP 1788 H=1788:00 +CPU= 58.646
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001788
  02:10:48 STEP 1789 H=1789:00 +CPU= 34.315
  02:11:25 STEP 1790 H=1790:00 +CPU= 35.582
  02:12:20 STEP 1791 H=1791:00 +CPU= 53.356
  02:12:56 STEP 1792 H=1792:00 +CPU= 35.369
  02:13:32 STEP 1793 H=1793:00 +CPU= 34.444
  02:14:28 STEP 1794 H=1794:00 +CPU= 54.158
  02:15:04 STEP 1795 H=1795:00 +CPU= 34.702
  02:15:42 STEP 1796 H=1796:00 +CPU= 35.269
  02:16:38 STEP 1797 H=1797:00 +CPU= 53.459
  02:17:14 STEP 1798 H=1798:00 +CPU= 34.905
  02:17:57 STEP 1799 H=1799:00 +CPU= 35.773
  02:20:24 STEP 1800 H=1800:00 +CPU= 67.804
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_74.zip
Uploading the intermediate file: upload_file_74.zip
  02:21:35 STEP 1801 H=1801:00 +CPU= 35.044
  02:22:42 STEP 1802 H=1802:00 +CPU= 35.700
  02:25:27 STEP 1803 H=1803:00 +CPU= 53.463
  02:26:08 STEP 1804 H=1804:00 +CPU= 35.624
  02:26:51 STEP 1805 H=1805:00 +CPU= 34.271
  02:27:51 STEP 1806 H=1806:00 +CPU= 53.583
  02:28:29 STEP 1807 H=1807:00 +CPU= 34.572
  02:30:35 STEP 1808 H=1808:00 +CPU= 36.522
  02:31:33 STEP 1809 H=1809:00 +CPU= 53.066
  02:32:12 STEP 1810 H=1810:00 +CPU= 36.792
  02:32:50 STEP 1811 H=1811:00 +CPU= 35.650
  02:34:56 STEP 1812 H=1812:00 +CPU= 60.231
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001812
  02:35:56 STEP 1813 H=1813:00 +CPU= 34.397
  02:37:47 STEP 1814 H=1814:00 +CPU= 37.377
  02:38:46 STEP 1815 H=1815:00 +CPU= 53.045
  02:39:24 STEP 1816 H=1816:00 +CPU= 35.562
  02:40:01 STEP 1817 H=1817:00 +CPU= 35.036
  02:41:04 STEP 1818 H=1818:00 +CPU= 52.258
  02:41:44 STEP 1819 H=1819:00 +CPU= 33.661
  02:42:24 STEP 1820 H=1820:00 +CPU= 35.871
  02:43:32 STEP 1821 H=1821:00 +CPU= 52.670
  02:44:08 STEP 1822 H=1822:00 +CPU= 34.701
  02:44:47 STEP 1823 H=1823:00 +CPU= 36.210
  02:46:22 STEP 1824 H=1824:00 +CPU= 63.886
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_75.zip
Uploading the intermediate file: upload_file_75.zip
Uploading trickle at timestep: 6566400
  02:47:08 STEP 1825 H=1825:00 +CPU= 33.648
  02:47:49 STEP 1826 H=1826:00 +CPU= 35.744
  02:49:00 STEP 1827 H=1827:00 +CPU= 53.817
  02:50:41 STEP 1828 H=1828:00 +CPU= 37.027
  02:51:20 STEP 1829 H=1829:00 +CPU= 34.906
  02:52:23 STEP 1830 H=1830:00 +CPU= 52.826
  02:53:01 STEP 1831 H=1831:00 +CPU= 34.008
  02:53:46 STEP 1832 H=1832:00 +CPU= 35.975
  02:55:04 STEP 1833 H=1833:00 +CPU= 52.786
  02:55:44 STEP 1834 H=1834:00 +CPU= 35.463
  02:56:20 STEP 1835 H=1835:00 +CPU= 34.740
  02:58:51 STEP 1836 H=1836:00 +CPU= 58.754
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001836
  03:00:42 STEP 1837 H=1837:00 +CPU= 37.168
  03:01:23 STEP 1838 H=1838:00 +CPU= 34.742
  03:02:25 STEP 1839 H=1839:00 +CPU= 54.177
  03:03:11 STEP 1840 H=1840:00 +CPU= 37.762
  03:05:23 STEP 1841 H=1841:00 +CPU= 34.658
  03:06:21 STEP 1842 H=1842:00 +CPU= 53.640
  03:06:57 STEP 1843 H=1843:00 +CPU= 34.181
  03:08:03 STEP 1844 H=1844:00 +CPU= 35.361
  03:09:06 STEP 1845 H=1845:00 +CPU= 53.511
  03:10:07 STEP 1846 H=1846:00 +CPU= 35.345
  03:11:52 STEP 1847 H=1847:00 +CPU= 36.455
  03:13:34 STEP 1848 H=1848:00 +CPU= 64.859
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_76.zip
Uploading the intermediate file: upload_file_76.zip
  03:14:11 STEP 1849 H=1849:00 +CPU= 34.964
  03:14:57 STEP 1850 H=1850:00 +CPU= 34.712
  03:16:58 STEP 1851 H=1851:00 +CPU= 53.933
  03:17:40 STEP 1852 H=1852:00 +CPU= 35.945
  03:20:20 STEP 1853 H=1853:00 +CPU= 35.161
  03:21:24 STEP 1854 H=1854:00 +CPU= 51.021
  03:24:17 STEP 1855 H=1855:00 +CPU= 36.351
  03:25:05 STEP 1856 H=1856:00 +CPU= 34.700
  03:28:31 STEP 1857 H=1857:00 +CPU= 54.959
  03:29:43 STEP 1858 H=1858:00 +CPU= 36.248
  03:30:20 STEP 1859 H=1859:00 +CPU= 34.573
  03:33:00 STEP 1860 H=1860:00 +CPU= 59.696
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001860
  03:33:40 STEP 1861 H=1861:00 +CPU= 35.072
  03:34:32 STEP 1862 H=1862:00 +CPU= 34.484
  03:35:28 STEP 1863 H=1863:00 +CPU= 52.856
  03:36:05 STEP 1864 H=1864:00 +CPU= 35.508
  03:36:40 STEP 1865 H=1865:00 +CPU= 34.290
  03:37:35 STEP 1866 H=1866:00 +CPU= 53.680
  03:38:11 STEP 1867 H=1867:00 +CPU= 34.233
  03:38:45 STEP 1868 H=1868:00 +CPU= 33.271
  03:40:36 STEP 1869 H=1869:00 +CPU= 53.789
  03:41:15 STEP 1870 H=1870:00 +CPU= 35.385
  03:41:53 STEP 1871 H=1871:00 +CPU= 36.066
  03:43:31 STEP 1872 H=1872:00 +CPU= 64.045
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_77.zip
Uploading the intermediate file: upload_file_77.zip
  03:44:11 STEP 1873 H=1873:00 +CPU= 35.098
  03:44:48 STEP 1874 H=1874:00 +CPU= 34.510
  03:45:43 STEP 1875 H=1875:00 +CPU= 53.288
  03:46:21 STEP 1876 H=1876:00 +CPU= 35.772
  03:46:55 STEP 1877 H=1877:00 +CPU= 33.695
  03:47:52 STEP 1878 H=1878:00 +CPU= 54.191
  03:48:30 STEP 1879 H=1879:00 +CPU= 34.749
  03:50:01 STEP 1880 H=1880:00 +CPU= 36.256
  03:51:00 STEP 1881 H=1881:00 +CPU= 52.571
  03:51:37 STEP 1882 H=1882:00 +CPU= 35.534
  03:52:17 STEP 1883 H=1883:00 +CPU= 34.098
  03:53:27 STEP 1884 H=1884:00 +CPU= 58.640
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001884
  03:54:08 STEP 1885 H=1885:00 +CPU= 36.797
  03:55:36 STEP 1886 H=1886:00 +CPU= 35.793
  03:56:33 STEP 1887 H=1887:00 +CPU= 53.340
  03:57:09 STEP 1888 H=1888:00 +CPU= 35.248
  03:57:46 STEP 1889 H=1889:00 +CPU= 34.331
  03:58:42 STEP 1890 H=1890:00 +CPU= 53.879
  03:59:17 STEP 1891 H=1891:00 +CPU= 34.072
  04:02:55 STEP 1892 H=1892:00 +CPU= 33.268
  04:03:54 STEP 1893 H=1893:00 +CPU= 53.148
  04:04:31 STEP 1894 H=1894:00 +CPU= 35.174
  04:05:22 STEP 1895 H=1895:00 +CPU= 34.886
  04:07:02 STEP 1896 H=1896:00 +CPU= 64.518
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_78.zip
Uploading the intermediate file: upload_file_78.zip
  04:07:42 STEP 1897 H=1897:00 +CPU= 34.153
  04:08:25 STEP 1898 H=1898:00 +CPU= 36.041
  04:09:21 STEP 1899 H=1899:00 +CPU= 53.312
  04:10:08 STEP 1900 H=1900:00 +CPU= 35.544
  04:10:44 STEP 1901 H=1901:00 +CPU= 35.032
  04:11:39 STEP 1902 H=1902:00 +CPU= 53.727
  04:12:17 STEP 1903 H=1903:00 +CPU= 34.099
  04:12:56 STEP 1904 H=1904:00 +CPU= 35.545
  04:13:51 STEP 1905 H=1905:00 +CPU= 53.220
  04:14:27 STEP 1906 H=1906:00 +CPU= 34.831
  04:15:02 STEP 1907 H=1907:00 +CPU= 35.078
  04:16:11 STEP 1908 H=1908:00 +CPU= 58.305
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001908
  04:16:47 STEP 1909 H=1909:00 +CPU= 34.267
  04:17:24 STEP 1910 H=1910:00 +CPU= 35.616
  04:18:33 STEP 1911 H=1911:00 +CPU= 53.032
  04:19:11 STEP 1912 H=1912:00 +CPU= 34.683
  04:19:47 STEP 1913 H=1913:00 +CPU= 34.815
  04:20:53 STEP 1914 H=1914:00 +CPU= 53.533
  04:21:28 STEP 1915 H=1915:00 +CPU= 33.636
  04:23:43 STEP 1916 H=1916:00 +CPU= 33.767
  04:25:11 STEP 1917 H=1917:00 +CPU= 55.437
  04:26:10 STEP 1918 H=1918:00 +CPU= 35.197
  04:27:00 STEP 1919 H=1919:00 +CPU= 35.345
  04:29:43 STEP 1920 H=1920:00 +CPU= 64.914
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328_79.zip
Uploading the intermediate file: upload_file_79.zip
  04:32:43 STEP 1921 H=1921:00 +CPU= 36.540
  04:33:26 STEP 1922 H=1922:00 +CPU= 34.268
  04:35:49 STEP 1923 H=1923:00 +CPU= 54.472
  04:37:52 STEP 1924 H=1924:00 +CPU= 36.825
  04:38:29 STEP 1925 H=1925:00 +CPU= 34.607
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0186
(argv4) batchid: 956
(argv5) wuid: 12172830
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/5
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12172830
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e40be6eb0a76bc84586cd977698623f1 to: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0186_1987050100_123_956_12172830.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/5/oifs_43r3_ps_0186_1987050100_123_956_12172830.zip
ic_ancil_file: ic_ancil_12172830
ifsdata_file: ifsdata_12172830
climate_data_file: clim_data_12172830
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_177aa6b6e839ad18fe54f412db351241 to: /var/lib/boinc-client/slots/5/ic_ancil_12172830.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/5/ic_ancil_12172830.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_c6183187ac7bd66d3acab98b28087d16 to: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12172830.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/5/ifsdata/ifsdata_12172830.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/5/159l_2/clim_data_12172830.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/5/159l_2/clim_data_12172830.zip
Checking for progress XML file: /var/lib/boinc-client/slots/5/progress_file_12172830.xml
Opened progress file ok : /var/lib/boinc-client/slots/5/progress_file_12172830.xml
-- Model is restarting --
Adjusting last_iter, 1926, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/5/progress_file_12172830.xml
last_cpu_time: 79752
upload_file_number: 80
last_iter: 1919
last_upload: 6912000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0186_1987050100_123_956_12172830_1_r1782345328
The child process has been launched with process id: 8557
Executing the command: /var/lib/boinc-client/slots/5/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:backup:1:1:8557:8557] [20230103:050344:1672718624:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:05:35 STEP 1920 H=1920:00 +CPU= 59.897
  05:14:13 STEP 1921 H=1921:00 +CPU= 42.549
  05:15:50 STEP 1922 H=1922:00 +CPU= 33.561
  05:16:44 STEP 1923 H=1923:00 +CPU= 50.616
  05:17:26 STEP 1924 H=1924:00 +CPU= 32.776
  05:18:53 STEP 1925 H=1925:00 +CPU= 33.163
  05:19:51 STEP 1926 H=1926:00 +CPU= 53.259
  05:21:02 STEP 1927 H=1927:00 +CPU= 33.860
  05:23:43 STEP 1928 H=1928:00 +CPU= 36.220
  05:28:04 STEP 1929 H=1929:00 +CPU= 55.654
  05:28:50 STEP 1930 H=1930:00 +CPU= 35.756
  05:29:58 STEP 1931 H=1931:00 +CPU= 33.783
  05:30:59 STEP 1932 H=1932:00 +CPU= 55.602
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/5/ICMUAhq0f+001932
  05:31:32 STEP 1933 H=1933:00 +CPU= 32.612
  05:37:40 STEP 1934 H=1934:00 +CPU= 38.582
  05:40:58 STEP 1935 H=1935:00 +CPU= 56.900
  05:44:05 STEP 1936 H=1936:00 +CPU= 37.704
  05:46:41 STEP 1937 H=1937:00 +CPU= 34.062
  05:47:34 STEP 1938 H=1938:00 +CPU= 51.672
  05:48:12 STEP 1939 H=1939:00 +CPU= 35.139
  05:48:49 STEP 1940 H=1940:00 +CPU= 35.134
  05:50:04 STEP 1941 H=1941:00 +CPU= 53.137
  05:50:41 STEP 1942 H=1942:00 +CPU= 35.389
  05:51:34 STEP 1943 H=1943:00 +CPU= 35.123
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:06:30 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 6,566,400 75,507 0.0115
09 May 2023 16:06:30 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 6,566,400 75,507 0.0115
09 May 2023 16:06:07 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 4,579,200 50,804 0.0111
09 May 2023 16:05:57 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 3,715,200 40,754 0.0110
09 May 2023 16:05:47 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 2,851,200 30,589 0.0107
09 May 2023 16:05:28 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 1,728,000 17,708 0.0102
09 May 2023 16:05:19 1499440 22264666 oifs_43r3_ps_0186_1987050100_123_956_12172830_1 864,000 8,890 0.0103


©2024 cpdn.org