climateprediction.net home page
Task 22265322

Task 22265322

Name oifs_43r3_ps_0719_1994050100_123_963_12180363_0
Workunit 12180363
Created 22 Dec 2022, 6:28:28 UTC
Sent 8 Jan 2023, 12:33:40 UTC
Report deadline 7 Feb 2023, 12:33:40 UTC
Received 9 Jan 2023, 14:42:32 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1463165
Run time 12 hours 39 min 29 sec
CPU time 12 hours 38 min 12 sec
Validate state Invalid
Credit 4,838.40
Device peak FLOPS 4.74 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,798.38 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,171.95 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
0 +CPU= 19.582
  23:06:04 STEP 1515 H=1515:00 +CPU= 30.581
  23:06:24 STEP 1516 H=1516:00 +CPU= 19.572
  23:06:43 STEP 1517 H=1517:00 +CPU= 18.956
  23:07:14 STEP 1518 H=1518:00 +CPU= 31.234
  23:07:33 STEP 1519 H=1519:00 +CPU= 18.984
  23:07:53 STEP 1520 H=1520:00 +CPU= 19.602
  23:08:23 STEP 1521 H=1521:00 +CPU= 30.562
  23:08:43 STEP 1522 H=1522:00 +CPU= 19.601
  23:09:02 STEP 1523 H=1523:00 +CPU= 18.965
  23:09:35 STEP 1524 H=1524:00 +CPU= 33.760
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001524
  23:09:54 STEP 1525 H=1525:00 +CPU= 18.999
  23:10:14 STEP 1526 H=1526:00 +CPU= 19.595
  23:10:45 STEP 1527 H=1527:00 +CPU= 30.550
  23:11:04 STEP 1528 H=1528:00 +CPU= 19.584
  23:11:23 STEP 1529 H=1529:00 +CPU= 18.949
  23:11:54 STEP 1530 H=1530:00 +CPU= 31.233
  23:12:13 STEP 1531 H=1531:00 +CPU= 18.947
  23:12:33 STEP 1532 H=1532:00 +CPU= 19.546
  23:13:03 STEP 1533 H=1533:00 +CPU= 30.519
  23:13:23 STEP 1534 H=1534:00 +CPU= 19.559
  23:13:42 STEP 1535 H=1535:00 +CPU= 19.464
  23:14:18 STEP 1536 H=1536:00 +CPU= 35.412
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_63.zip
Uploading the intermediate file: upload_file_63.zip
  23:14:37 STEP 1537 H=1537:00 +CPU= 18.928
  23:14:56 STEP 1538 H=1538:00 +CPU= 19.500
  23:15:27 STEP 1539 H=1539:00 +CPU= 30.422
  23:15:46 STEP 1540 H=1540:00 +CPU= 19.492
  23:16:05 STEP 1541 H=1541:00 +CPU= 18.891
  23:16:36 STEP 1542 H=1542:00 +CPU= 31.184
  23:16:55 STEP 1543 H=1543:00 +CPU= 18.901
  23:17:15 STEP 1544 H=1544:00 +CPU= 19.531
  23:17:45 STEP 1545 H=1545:00 +CPU= 30.482
  23:18:05 STEP 1546 H=1546:00 +CPU= 19.546
  23:18:24 STEP 1547 H=1547:00 +CPU= 18.926
  23:18:57 STEP 1548 H=1548:00 +CPU= 33.709
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001548
  23:19:16 STEP 1549 H=1549:00 +CPU= 18.922
  23:19:36 STEP 1550 H=1550:00 +CPU= 19.540
  23:20:06 STEP 1551 H=1551:00 +CPU= 30.476
  23:20:26 STEP 1552 H=1552:00 +CPU= 19.543
  23:20:45 STEP 1553 H=1553:00 +CPU= 18.895
  23:21:16 STEP 1554 H=1554:00 +CPU= 31.173
  23:21:35 STEP 1555 H=1555:00 +CPU= 18.876
  23:21:54 STEP 1556 H=1556:00 +CPU= 19.490
  23:22:25 STEP 1557 H=1557:00 +CPU= 30.467
  23:22:44 STEP 1558 H=1558:00 +CPU= 19.495
  23:23:04 STEP 1559 H=1559:00 +CPU= 19.409
  23:23:39 STEP 1560 H=1560:00 +CPU= 35.518
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_64.zip
Uploading the intermediate file: upload_file_64.zip
  23:23:58 STEP 1561 H=1561:00 +CPU= 19.005
  23:24:18 STEP 1562 H=1562:00 +CPU= 19.526
  23:24:48 STEP 1563 H=1563:00 +CPU= 30.501
  23:25:08 STEP 1564 H=1564:00 +CPU= 19.521
  23:25:27 STEP 1565 H=1565:00 +CPU= 18.925
  23:25:58 STEP 1566 H=1566:00 +CPU= 31.211
  23:26:17 STEP 1567 H=1567:00 +CPU= 18.931
  23:26:37 STEP 1568 H=1568:00 +CPU= 19.551
  23:27:07 STEP 1569 H=1569:00 +CPU= 30.498
  23:27:27 STEP 1570 H=1570:00 +CPU= 19.577
  23:27:46 STEP 1571 H=1571:00 +CPU= 18.952
  23:28:19 STEP 1572 H=1572:00 +CPU= 33.697
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001572
  23:28:38 STEP 1573 H=1573:00 +CPU= 18.961
  23:28:58 STEP 1574 H=1574:00 +CPU= 19.559
  23:29:28 STEP 1575 H=1575:00 +CPU= 30.491
  23:29:48 STEP 1576 H=1576:00 +CPU= 19.548
  23:30:07 STEP 1577 H=1577:00 +CPU= 18.937
  23:30:38 STEP 1578 H=1578:00 +CPU= 31.177
  23:30:57 STEP 1579 H=1579:00 +CPU= 18.911
  23:31:16 STEP 1580 H=1580:00 +CPU= 19.509
  23:31:47 STEP 1581 H=1581:00 +CPU= 30.447
  23:32:06 STEP 1582 H=1582:00 +CPU= 19.493
  23:32:26 STEP 1583 H=1583:00 +CPU= 19.419
  23:33:01 STEP 1584 H=1584:00 +CPU= 35.414
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_65.zip
Uploading the intermediate file: upload_file_65.zip
  23:33:20 STEP 1585 H=1585:00 +CPU= 18.985
  23:33:40 STEP 1586 H=1586:00 +CPU= 19.547
  23:34:10 STEP 1587 H=1587:00 +CPU= 30.491
  23:34:30 STEP 1588 H=1588:00 +CPU= 19.546
  23:34:49 STEP 1589 H=1589:00 +CPU= 18.927
  23:35:20 STEP 1590 H=1590:00 +CPU= 31.173
  23:35:39 STEP 1591 H=1591:00 +CPU= 18.931
  23:35:58 STEP 1592 H=1592:00 +CPU= 19.535
  23:36:29 STEP 1593 H=1593:00 +CPU= 30.473
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0719
(argv4) batchid: 963
(argv5) wuid: 12180363
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a2bc85563e51f609d7007740b857fcfc to: /var/lib/boinc/slots/1/oifs_43r3_ps_0719_1994050100_123_963_12180363.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0719_1994050100_123_963_12180363.zip
ic_ancil_file: ic_ancil_12180363
ifsdata_file: ifsdata_12180363
climate_data_file: clim_data_12180363
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7484f44e5ea4c44a183273b3075bc914 to: /var/lib/boinc/slots/1/ic_ancil_12180363.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12180363.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9459aeec7801ffb63d88f651a3da602d to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12180363.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12180363.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12180363.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12180363.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12180363.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12180363.xml
-- Model is restarting --
Adjusting last_iter, 1594, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12180363.xml
last_cpu_time: 38149
upload_file_number: 66
last_iter: 1583
last_upload: 5702400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883
The child process has been launched with process id: 20473
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:20473:20473] [20230109:013957:1673253597:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:40:32 STEP 1584 H=1584:00 +CPU= 33.698
  01:40:51 STEP 1585 H=1585:00 +CPU= 18.850
  01:41:11 STEP 1586 H=1586:00 +CPU= 19.469
  01:41:41 STEP 1587 H=1587:00 +CPU= 30.412
  01:42:00 STEP 1588 H=1588:00 +CPU= 19.490
  01:42:19 STEP 1589 H=1589:00 +CPU= 18.868
  01:42:51 STEP 1590 H=1590:00 +CPU= 31.165
  01:43:09 STEP 1591 H=1591:00 +CPU= 18.896
  01:43:29 STEP 1592 H=1592:00 +CPU= 19.540
  01:43:59 STEP 1593 H=1593:00 +CPU= 30.456
  01:44:19 STEP 1594 H=1594:00 +CPU= 19.537
  01:44:38 STEP 1595 H=1595:00 +CPU= 18.924
  01:45:12 STEP 1596 H=1596:00 +CPU= 33.718
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001596
  01:45:31 STEP 1597 H=1597:00 +CPU= 18.933
  01:45:50 STEP 1598 H=1598:00 +CPU= 19.558
  01:46:21 STEP 1599 H=1599:00 +CPU= 30.505
  01:46:40 STEP 1600 H=1600:00 +CPU= 19.567
  01:46:59 STEP 1601 H=1601:00 +CPU= 18.925
  01:47:30 STEP 1602 H=1602:00 +CPU= 31.183
  01:47:49 STEP 1603 H=1603:00 +CPU= 18.890
  01:48:09 STEP 1604 H=1604:00 +CPU= 19.497
  01:48:39 STEP 1605 H=1605:00 +CPU= 30.425
  01:48:59 STEP 1606 H=1606:00 +CPU= 19.470
  01:49:18 STEP 1607 H=1607:00 +CPU= 19.384
  01:49:54 STEP 1608 H=1608:00 +CPU= 35.817
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_66.zip
Uploading the intermediate file: upload_file_66.zip
  01:50:13 STEP 1609 H=1609:00 +CPU= 19.023
  01:50:32 STEP 1610 H=1610:00 +CPU= 19.613
  01:51:03 STEP 1611 H=1611:00 +CPU= 30.588
  01:51:23 STEP 1612 H=1612:00 +CPU= 19.631
  01:51:42 STEP 1613 H=1613:00 +CPU= 19.013
  01:52:13 STEP 1614 H=1614:00 +CPU= 31.315
  01:52:32 STEP 1615 H=1615:00 +CPU= 19.015
  01:52:52 STEP 1616 H=1616:00 +CPU= 19.650
  01:53:22 STEP 1617 H=1617:00 +CPU= 30.614
  01:53:42 STEP 1618 H=1618:00 +CPU= 19.641
  01:54:01 STEP 1619 H=1619:00 +CPU= 19.050
  01:54:35 STEP 1620 H=1620:00 +CPU= 33.839
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001620
  01:54:54 STEP 1621 H=1621:00 +CPU= 19.043
  01:55:14 STEP 1622 H=1622:00 +CPU= 19.682
  01:55:44 STEP 1623 H=1623:00 +CPU= 30.604
  01:56:04 STEP 1624 H=1624:00 +CPU= 19.629
  01:56:23 STEP 1625 H=1625:00 +CPU= 19.003
  01:56:54 STEP 1626 H=1626:00 +CPU= 31.289
  01:57:13 STEP 1627 H=1627:00 +CPU= 18.997
  01:57:33 STEP 1628 H=1628:00 +CPU= 19.606
  01:58:03 STEP 1629 H=1629:00 +CPU= 30.558
  01:58:23 STEP 1630 H=1630:00 +CPU= 19.584
  01:58:42 STEP 1631 H=1631:00 +CPU= 19.517
  01:59:18 STEP 1632 H=1632:00 +CPU= 35.392
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_67.zip
Uploading the intermediate file: upload_file_67.zip
  01:59:37 STEP 1633 H=1633:00 +CPU= 18.961
  01:59:56 STEP 1634 H=1634:00 +CPU= 19.508
  02:00:27 STEP 1635 H=1635:00 +CPU= 30.444
  02:00:46 STEP 1636 H=1636:00 +CPU= 19.509
  02:01:05 STEP 1637 H=1637:00 +CPU= 18.904
  02:01:36 STEP 1638 H=1638:00 +CPU= 31.168
  02:01:55 STEP 1639 H=1639:00 +CPU= 18.913
  02:02:15 STEP 1640 H=1640:00 +CPU= 19.546
  02:02:45 STEP 1641 H=1641:00 +CPU= 30.480
  02:03:05 STEP 1642 H=1642:00 +CPU= 19.549
  02:03:24 STEP 1643 H=1643:00 +CPU= 18.918
  02:03:57 STEP 1644 H=1644:00 +CPU= 33.679
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001644
  02:04:16 STEP 1645 H=1645:00 +CPU= 18.922
  02:04:36 STEP 1646 H=1646:00 +CPU= 19.539
  02:05:06 STEP 1647 H=1647:00 +CPU= 30.451
  02:05:26 STEP 1648 H=1648:00 +CPU= 19.524
  02:05:45 STEP 1649 H=1649:00 +CPU= 18.908
  02:06:16 STEP 1650 H=1650:00 +CPU= 31.158
  02:06:35 STEP 1651 H=1651:00 +CPU= 18.891
  02:06:54 STEP 1652 H=1652:00 +CPU= 19.498
  02:07:25 STEP 1653 H=1653:00 +CPU= 30.455
  02:07:44 STEP 1654 H=1654:00 +CPU= 19.512
  02:08:04 STEP 1655 H=1655:00 +CPU= 19.430
  02:08:39 STEP 1656 H=1656:00 +CPU= 35.542
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_68.zip
Uploading the intermediate file: upload_file_68.zip
  02:08:58 STEP 1657 H=1657:00 +CPU= 19.025
  02:09:18 STEP 1658 H=1658:00 +CPU= 19.597
  02:09:49 STEP 1659 H=1659:00 +CPU= 30.592
  02:10:08 STEP 1660 H=1660:00 +CPU= 19.615
  02:10:27 STEP 1661 H=1661:00 +CPU= 18.986
  02:10:58 STEP 1662 H=1662:00 +CPU= 31.322
  02:11:17 STEP 1663 H=1663:00 +CPU= 19.002
  02:11:37 STEP 1664 H=1664:00 +CPU= 19.573
  02:12:08 STEP 1665 H=1665:00 +CPU= 30.528
  02:12:27 STEP 1666 H=1666:00 +CPU= 19.597
  02:12:46 STEP 1667 H=1667:00 +CPU= 18.983
  02:13:20 STEP 1668 H=1668:00 +CPU= 33.745
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001668
  02:13:39 STEP 1669 H=1669:00 +CPU= 18.962
  02:13:58 STEP 1670 H=1670:00 +CPU= 19.579
  02:14:29 STEP 1671 H=1671:00 +CPU= 30.535
  02:14:49 STEP 1672 H=1672:00 +CPU= 19.543
  02:15:07 STEP 1673 H=1673:00 +CPU= 18.908
  02:15:39 STEP 1674 H=1674:00 +CPU= 31.198
  02:15:58 STEP 1675 H=1675:00 +CPU= 18.921
  02:16:17 STEP 1676 H=1676:00 +CPU= 19.508
  02:16:48 STEP 1677 H=1677:00 +CPU= 30.483
  02:17:07 STEP 1678 H=1678:00 +CPU= 19.517
  02:17:27 STEP 1679 H=1679:00 +CPU= 19.430
  02:18:02 STEP 1680 H=1680:00 +CPU= 35.425
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_69.zip
Uploading the intermediate file: upload_file_69.zip
  02:18:21 STEP 1681 H=1681:00 +CPU= 18.992
  02:18:40 STEP 1682 H=1682:00 +CPU= 19.495
  02:19:11 STEP 1683 H=1683:00 +CPU= 30.449
  02:19:30 STEP 1684 H=1684:00 +CPU= 19.522
  02:19:49 STEP 1685 H=1685:00 +CPU= 18.894
  02:20:21 STEP 1686 H=1686:00 +CPU= 31.174
  02:20:39 STEP 1687 H=1687:00 +CPU= 18.923
  02:20:59 STEP 1688 H=1688:00 +CPU= 19.548
  02:21:30 STEP 1689 H=1689:00 +CPU= 30.484
  02:21:49 STEP 1690 H=1690:00 +CPU= 19.558
  02:22:08 STEP 1691 H=1691:00 +CPU= 18.932
  02:22:42 STEP 1692 H=1692:00 +CPU= 33.702
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001692
  02:23:01 STEP 1693 H=1693:00 +CPU= 18.936
  02:23:20 STEP 1694 H=1694:00 +CPU= 19.550
  02:23:51 STEP 1695 H=1695:00 +CPU= 30.466
  02:24:10 STEP 1696 H=1696:00 +CPU= 19.539
  02:24:29 STEP 1697 H=1697:00 +CPU= 18.905
  02:25:00 STEP 1698 H=1698:00 +CPU= 31.163
  02:25:19 STEP 1699 H=1699:00 +CPU= 18.892
  02:25:39 STEP 1700 H=1700:00 +CPU= 19.496
  02:26:09 STEP 1701 H=1701:00 +CPU= 30.430
  02:26:29 STEP 1702 H=1702:00 +CPU= 19.482
  02:26:48 STEP 1703 H=1703:00 +CPU= 19.401
  02:27:23 STEP 1704 H=1704:00 +CPU= 35.396
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_70.zip
Uploading the intermediate file: upload_file_70.zip
  02:27:42 STEP 1705 H=1705:00 +CPU= 18.983
  02:28:02 STEP 1706 H=1706:00 +CPU= 19.494
  02:28:32 STEP 1707 H=1707:00 +CPU= 30.405
  02:28:52 STEP 1708 H=1708:00 +CPU= 19.507
  02:29:11 STEP 1709 H=1709:00 +CPU= 18.919
  02:29:42 STEP 1710 H=1710:00 +CPU= 31.143
  02:30:01 STEP 1711 H=1711:00 +CPU= 18.918
  02:30:20 STEP 1712 H=1712:00 +CPU= 19.548
  02:30:51 STEP 1713 H=1713:00 +CPU= 30.467
  02:31:10 STEP 1714 H=1714:00 +CPU= 19.559
  02:31:29 STEP 1715 H=1715:00 +CPU= 18.928
  02:32:03 STEP 1716 H=1716:00 +CPU= 33.681
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001716
  02:32:22 STEP 1717 H=1717:00 +CPU= 18.931
  02:32:41 STEP 1718 H=1718:00 +CPU= 19.537
  02:33:12 STEP 1719 H=1719:00 +CPU= 30.430
  02:33:31 STEP 1720 H=1720:00 +CPU= 19.532
  02:33:50 STEP 1721 H=1721:00 +CPU= 18.892
  02:34:21 STEP 1722 H=1722:00 +CPU= 31.123
  02:34:40 STEP 1723 H=1723:00 +CPU= 18.878
  02:35:00 STEP 1724 H=1724:00 +CPU= 19.481
  02:35:30 STEP 1725 H=1725:00 +CPU= 30.385
  02:35:50 STEP 1726 H=1726:00 +CPU= 19.454
  02:36:09 STEP 1727 H=1727:00 +CPU= 19.382
  02:36:44 STEP 1728 H=1728:00 +CPU= 35.388
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_71.zip
Uploading the intermediate file: upload_file_71.zip
  02:37:03 STEP 1729 H=1729:00 +CPU= 18.935
  02:37:23 STEP 1730 H=1730:00 +CPU= 19.513
  02:37:53 STEP 1731 H=1731:00 +CPU= 30.438
  02:38:13 STEP 1732 H=1732:00 +CPU= 19.528
  02:38:32 STEP 1733 H=1733:00 +CPU= 18.915
  02:39:03 STEP 1734 H=1734:00 +CPU= 31.134
  02:39:22 STEP 1735 H=1735:00 +CPU= 18.918
  02:39:41 STEP 1736 H=1736:00 +CPU= 19.520
  02:40:12 STEP 1737 H=1737:00 +CPU= 30.403
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0719
(argv4) batchid: 963
(argv5) wuid: 12180363
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a2bc85563e51f609d7007740b857fcfc to: /var/lib/boinc/slots/1/oifs_43r3_ps_0719_1994050100_123_963_12180363.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0719_1994050100_123_963_12180363.zip
ic_ancil_file: ic_ancil_12180363
ifsdata_file: ifsdata_12180363
climate_data_file: clim_data_12180363
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7484f44e5ea4c44a183273b3075bc914 to: /var/lib/boinc/slots/1/ic_ancil_12180363.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12180363.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9459aeec7801ffb63d88f651a3da602d to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12180363.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12180363.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12180363.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12180363.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12180363.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12180363.xml
-- Model is restarting --
Adjusting last_iter, 1738, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12180363.xml
last_cpu_time: 41765
upload_file_number: 72
last_iter: 1727
last_upload: 6220800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883
The child process has been launched with process id: 21623
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:21623:21623] [20230109:043350:1673264030:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:34:25 STEP 1728 H=1728:00 +CPU= 33.730
  04:34:44 STEP 1729 H=1729:00 +CPU= 18.858
  04:35:04 STEP 1730 H=1730:00 +CPU= 19.464
  04:35:34 STEP 1731 H=1731:00 +CPU= 30.423
  04:35:54 STEP 1732 H=1732:00 +CPU= 19.484
  04:36:12 STEP 1733 H=1733:00 +CPU= 18.873
  04:36:44 STEP 1734 H=1734:00 +CPU= 31.131
  04:37:02 STEP 1735 H=1735:00 +CPU= 18.876
  04:37:22 STEP 1736 H=1736:00 +CPU= 19.492
  04:37:52 STEP 1737 H=1737:00 +CPU= 30.425
  04:38:12 STEP 1738 H=1738:00 +CPU= 19.510
  04:38:31 STEP 1739 H=1739:00 +CPU= 18.885
  04:39:04 STEP 1740 H=1740:00 +CPU= 33.648
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001740
  04:39:23 STEP 1741 H=1741:00 +CPU= 18.913
  04:39:43 STEP 1742 H=1742:00 +CPU= 19.531
  04:40:13 STEP 1743 H=1743:00 +CPU= 30.449
  04:40:33 STEP 1744 H=1744:00 +CPU= 19.485
  04:40:52 STEP 1745 H=1745:00 +CPU= 18.855
  04:41:23 STEP 1746 H=1746:00 +CPU= 31.109
  04:41:42 STEP 1747 H=1747:00 +CPU= 18.849
  04:42:01 STEP 1748 H=1748:00 +CPU= 19.460
  04:42:32 STEP 1749 H=1749:00 +CPU= 30.394
  04:42:51 STEP 1750 H=1750:00 +CPU= 19.453
  04:43:10 STEP 1751 H=1751:00 +CPU= 19.362
  04:43:46 STEP 1752 H=1752:00 +CPU= 35.786
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_72.zip
Uploading the intermediate file: upload_file_72.zip
  04:44:05 STEP 1753 H=1753:00 +CPU= 19.098
  04:44:25 STEP 1754 H=1754:00 +CPU= 19.593
  04:44:55 STEP 1755 H=1755:00 +CPU= 30.569
  04:45:15 STEP 1756 H=1756:00 +CPU= 19.571
  04:45:34 STEP 1757 H=1757:00 +CPU= 18.968
  04:46:05 STEP 1758 H=1758:00 +CPU= 31.229
  04:46:24 STEP 1759 H=1759:00 +CPU= 18.987
  04:46:44 STEP 1760 H=1760:00 +CPU= 19.612
  04:47:14 STEP 1761 H=1761:00 +CPU= 30.527
  04:47:34 STEP 1762 H=1762:00 +CPU= 19.605
  04:47:53 STEP 1763 H=1763:00 +CPU= 18.966
  04:48:27 STEP 1764 H=1764:00 +CPU= 33.737
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001764
  04:48:46 STEP 1765 H=1765:00 +CPU= 18.987
  04:49:05 STEP 1766 H=1766:00 +CPU= 19.602
  04:49:36 STEP 1767 H=1767:00 +CPU= 30.528
  04:49:55 STEP 1768 H=1768:00 +CPU= 19.572
  04:50:14 STEP 1769 H=1769:00 +CPU= 18.950
  04:50:46 STEP 1770 H=1770:00 +CPU= 31.226
  04:51:04 STEP 1771 H=1771:00 +CPU= 18.952
  04:51:24 STEP 1772 H=1772:00 +CPU= 19.538
  04:51:55 STEP 1773 H=1773:00 +CPU= 30.491
  04:52:14 STEP 1774 H=1774:00 +CPU= 19.533
  04:52:34 STEP 1775 H=1775:00 +CPU= 19.457
  04:53:09 STEP 1776 H=1776:00 +CPU= 35.415
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_73.zip
Uploading the intermediate file: upload_file_73.zip
  04:53:28 STEP 1777 H=1777:00 +CPU= 18.950
  04:53:47 STEP 1778 H=1778:00 +CPU= 19.517
  04:54:18 STEP 1779 H=1779:00 +CPU= 30.522
  04:54:37 STEP 1780 H=1780:00 +CPU= 19.544
  04:54:56 STEP 1781 H=1781:00 +CPU= 18.916
  04:55:28 STEP 1782 H=1782:00 +CPU= 31.246
  04:55:47 STEP 1783 H=1783:00 +CPU= 18.938
  04:56:06 STEP 1784 H=1784:00 +CPU= 19.576
  04:56:37 STEP 1785 H=1785:00 +CPU= 30.563
  04:56:56 STEP 1786 H=1786:00 +CPU= 19.580
  04:57:15 STEP 1787 H=1787:00 +CPU= 18.939
  04:57:49 STEP 1788 H=1788:00 +CPU= 33.755
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001788
  04:58:08 STEP 1789 H=1789:00 +CPU= 18.957
  04:58:28 STEP 1790 H=1790:00 +CPU= 19.565
  04:58:58 STEP 1791 H=1791:00 +CPU= 30.516
  04:59:18 STEP 1792 H=1792:00 +CPU= 19.551
  04:59:37 STEP 1793 H=1793:00 +CPU= 18.913
  05:00:08 STEP 1794 H=1794:00 +CPU= 31.221
  05:00:27 STEP 1795 H=1795:00 +CPU= 18.905
  05:00:46 STEP 1796 H=1796:00 +CPU= 19.519
  05:01:17 STEP 1797 H=1797:00 +CPU= 30.495
  05:01:36 STEP 1798 H=1798:00 +CPU= 19.494
  05:01:56 STEP 1799 H=1799:00 +CPU= 19.424
  05:02:31 STEP 1800 H=1800:00 +CPU= 35.512
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_74.zip
Uploading the intermediate file: upload_file_74.zip
  05:02:50 STEP 1801 H=1801:00 +CPU= 18.938
  05:03:10 STEP 1802 H=1802:00 +CPU= 19.532
  05:03:40 STEP 1803 H=1803:00 +CPU= 30.529
  05:04:00 STEP 1804 H=1804:00 +CPU= 19.545
  05:04:19 STEP 1805 H=1805:00 +CPU= 18.926
  05:04:50 STEP 1806 H=1806:00 +CPU= 31.266
  05:05:09 STEP 1807 H=1807:00 +CPU= 18.974
  05:05:28 STEP 1808 H=1808:00 +CPU= 19.599
  05:05:59 STEP 1809 H=1809:00 +CPU= 30.608
  05:06:19 STEP 1810 H=1810:00 +CPU= 19.627
  05:06:38 STEP 1811 H=1811:00 +CPU= 18.997
  05:07:12 STEP 1812 H=1812:00 +CPU= 33.794
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001812
  05:07:30 STEP 1813 H=1813:00 +CPU= 18.959
  05:07:50 STEP 1814 H=1814:00 +CPU= 19.575
  05:08:21 STEP 1815 H=1815:00 +CPU= 30.557
  05:08:40 STEP 1816 H=1816:00 +CPU= 19.540
  05:08:59 STEP 1817 H=1817:00 +CPU= 18.934
  05:09:30 STEP 1818 H=1818:00 +CPU= 31.242
  05:09:49 STEP 1819 H=1819:00 +CPU= 18.910
  05:10:09 STEP 1820 H=1820:00 +CPU= 19.521
  05:10:39 STEP 1821 H=1821:00 +CPU= 30.545
  05:10:59 STEP 1822 H=1822:00 +CPU= 19.507
  05:11:18 STEP 1823 H=1823:00 +CPU= 19.417
  05:11:54 STEP 1824 H=1824:00 +CPU= 35.518
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_75.zip
Uploading the intermediate file: upload_file_75.zip
  05:12:13 STEP 1825 H=1825:00 +CPU= 18.960
  05:12:32 STEP 1826 H=1826:00 +CPU= 19.531
  05:13:03 STEP 1827 H=1827:00 +CPU= 30.571
  05:13:22 STEP 1828 H=1828:00 +CPU= 19.566
  05:13:41 STEP 1829 H=1829:00 +CPU= 18.952
  05:14:13 STEP 1830 H=1830:00 +CPU= 31.266
  05:14:32 STEP 1831 H=1831:00 +CPU= 18.975
  05:14:51 STEP 1832 H=1832:00 +CPU= 19.565
  05:15:22 STEP 1833 H=1833:00 +CPU= 30.574
  05:15:41 STEP 1834 H=1834:00 +CPU= 19.567
  05:16:00 STEP 1835 H=1835:00 +CPU= 18.958
  05:16:34 STEP 1836 H=1836:00 +CPU= 33.757
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001836
  05:16:53 STEP 1837 H=1837:00 +CPU= 18.959
  05:17:13 STEP 1838 H=1838:00 +CPU= 19.575
  05:17:43 STEP 1839 H=1839:00 +CPU= 30.537
  05:18:03 STEP 1840 H=1840:00 +CPU= 19.548
  05:18:22 STEP 1841 H=1841:00 +CPU= 18.920
  05:18:53 STEP 1842 H=1842:00 +CPU= 31.226
  05:19:12 STEP 1843 H=1843:00 +CPU= 18.912
  05:19:31 STEP 1844 H=1844:00 +CPU= 19.529
  05:20:02 STEP 1845 H=1845:00 +CPU= 30.533
  05:20:21 STEP 1846 H=1846:00 +CPU= 19.528
  05:20:41 STEP 1847 H=1847:00 +CPU= 19.425
  05:21:16 STEP 1848 H=1848:00 +CPU= 35.502
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_76.zip
Uploading the intermediate file: upload_file_76.zip
  05:21:35 STEP 1849 H=1849:00 +CPU= 18.981
  05:21:55 STEP 1850 H=1850:00 +CPU= 19.533
  05:22:25 STEP 1851 H=1851:00 +CPU= 30.541
  05:22:45 STEP 1852 H=1852:00 +CPU= 19.543
  05:23:04 STEP 1853 H=1853:00 +CPU= 18.945
  05:23:35 STEP 1854 H=1854:00 +CPU= 31.236
  05:23:54 STEP 1855 H=1855:00 +CPU= 18.949
  05:24:14 STEP 1856 H=1856:00 +CPU= 19.572
  05:24:44 STEP 1857 H=1857:00 +CPU= 30.536
  05:25:04 STEP 1858 H=1858:00 +CPU= 19.575
  05:25:23 STEP 1859 H=1859:00 +CPU= 18.957
  05:25:56 STEP 1860 H=1860:00 +CPU= 33.744
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001860
  05:26:15 STEP 1861 H=1861:00 +CPU= 18.962
  05:26:35 STEP 1862 H=1862:00 +CPU= 19.589
  05:27:06 STEP 1863 H=1863:00 +CPU= 30.559
  05:27:25 STEP 1864 H=1864:00 +CPU= 19.590
  05:27:44 STEP 1865 H=1865:00 +CPU= 18.962
  05:28:15 STEP 1866 H=1866:00 +CPU= 31.247
  05:28:34 STEP 1867 H=1867:00 +CPU= 18.952
  05:28:54 STEP 1868 H=1868:00 +CPU= 19.544
  05:29:24 STEP 1869 H=1869:00 +CPU= 30.520
  05:29:44 STEP 1870 H=1870:00 +CPU= 19.516
  05:30:03 STEP 1871 H=1871:00 +CPU= 19.445
  05:30:39 STEP 1872 H=1872:00 +CPU= 35.508
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883_77.zip
Uploading the intermediate file: upload_file_77.zip
  05:30:58 STEP 1873 H=1873:00 +CPU= 18.983
  05:31:17 STEP 1874 H=1874:00 +CPU= 19.540
  05:31:48 STEP 1875 H=1875:00 +CPU= 30.517
  05:32:08 STEP 1876 H=1876:00 +CPU= 19.542
  05:32:26 STEP 1877 H=1877:00 +CPU= 18.931
  05:32:58 STEP 1878 H=1878:00 +CPU= 31.266
  05:33:17 STEP 1879 H=1879:00 +CPU= 18.984
  05:33:36 STEP 1880 H=1880:00 +CPU= 19.581
  05:34:07 STEP 1881 H=1881:00 +CPU= 30.573
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0719
(argv4) batchid: 963
(argv5) wuid: 12180363
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12180363
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a2bc85563e51f609d7007740b857fcfc to: /var/lib/boinc/slots/1/oifs_43r3_ps_0719_1994050100_123_963_12180363.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0719_1994050100_123_963_12180363.zip
ic_ancil_file: ic_ancil_12180363
ifsdata_file: ifsdata_12180363
climate_data_file: clim_data_12180363
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7484f44e5ea4c44a183273b3075bc914 to: /var/lib/boinc/slots/1/ic_ancil_12180363.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12180363.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9459aeec7801ffb63d88f651a3da602d to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12180363.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12180363.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12180363.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12180363.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12180363.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12180363.xml
-- Model is restarting --
Adjusting last_iter, 1882, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12180363.xml
last_cpu_time: 45381
upload_file_number: 78
last_iter: 1871
last_upload: 6739200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0719_1994050100_123_963_12180363_0_r1401787883
The child process has been launched with process id: 22653
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:22653:22653] [20230109:073907:1673275147:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:39:43 STEP 1872 H=1872:00 +CPU= 33.715
  07:40:02 STEP 1873 H=1873:00 +CPU= 18.881
  07:40:21 STEP 1874 H=1874:00 +CPU= 19.488
  07:40:51 STEP 1875 H=1875:00 +CPU= 30.467
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:13:00 1463165 22265322 oifs_43r3_ps_0719_1994050100_123_963_12180363_0 3,456,000 22,485 0.0065
09 May 2023 16:12:57 1463165 22265322 oifs_43r3_ps_0719_1994050100_123_963_12180363_0 2,592,000 16,863 0.0065
09 May 2023 16:12:53 1463165 22265322 oifs_43r3_ps_0719_1994050100_123_963_12180363_0 1,728,000 11,251 0.0065
09 May 2023 16:12:50 1463165 22265322 oifs_43r3_ps_0719_1994050100_123_963_12180363_0 864,000 5,639 0.0065


©2024 cpdn.org