climateprediction.net home page
Task 22294042

Task 22294042

Name oifs_43r3_ps_0780_2020050100_123_989_12206424_1
Workunit 12206424
Created 28 Dec 2022, 12:33:50 UTC
Sent 28 Dec 2022, 13:59:22 UTC
Report deadline 27 Jan 2023, 13:59:22 UTC
Received 30 Dec 2022, 20:22:11 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1458811
Run time 16 hours 48 min 26 sec
CPU time 15 hours 57 min 29 sec
Validate state Invalid
Credit 1,209.60
Device peak FLOPS 3.17 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,069.17 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,333.63 MB
Stderr 
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000564
  07:37:51 STEP  565 H= 565:00 +CPU= 52.758
  07:38:45 STEP  566 H= 566:00 +CPU= 53.621
  07:40:15 STEP  567 H= 567:00 +CPU= 86.749
  07:41:09 STEP  568 H= 568:00 +CPU= 53.204
  07:42:02 STEP  569 H= 569:00 +CPU= 52.632
  07:43:29 STEP  570 H= 570:00 +CPU= 86.826
  07:44:22 STEP  571 H= 571:00 +CPU= 51.918
  07:45:17 STEP  572 H= 572:00 +CPU= 54.134
  07:46:44 STEP  573 H= 573:00 +CPU= 86.339
  07:47:39 STEP  574 H= 574:00 +CPU= 54.650
  07:48:34 STEP  575 H= 575:00 +CPU= 53.817
  07:50:18 STEP  576 H= 576:00 +CPU= 99.648
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_23.zip
Uploading the intermediate file: upload_file_23.zip
  07:51:12 STEP  577 H= 577:00 +CPU= 52.618
  07:52:06 STEP  578 H= 578:00 +CPU= 54.105
  07:53:32 STEP  579 H= 579:00 +CPU= 85.081
  07:54:27 STEP  580 H= 580:00 +CPU= 54.291
  07:55:20 STEP  581 H= 581:00 +CPU= 52.392
  07:56:47 STEP  582 H= 582:00 +CPU= 87.131
  07:57:39 STEP  583 H= 583:00 +CPU= 51.414
  07:58:33 STEP  584 H= 584:00 +CPU= 53.723
  07:59:59 STEP  585 H= 585:00 +CPU= 84.254
  08:00:54 STEP  586 H= 586:00 +CPU= 53.986
  08:01:47 STEP  587 H= 587:00 +CPU= 52.632
  08:03:22 STEP  588 H= 588:00 +CPU= 93.937
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000588
  08:04:15 STEP  589 H= 589:00 +CPU= 52.422
  08:05:09 STEP  590 H= 590:00 +CPU= 53.352
  08:06:35 STEP  591 H= 591:00 +CPU= 84.372
  08:07:29 STEP  592 H= 592:00 +CPU= 53.675
  08:08:22 STEP  593 H= 593:00 +CPU= 52.270
  08:09:50 STEP  594 H= 594:00 +CPU= 86.455
  08:10:42 STEP  595 H= 595:00 +CPU= 51.897
  08:11:37 STEP  596 H= 596:00 +CPU= 54.235
  08:13:03 STEP  597 H= 597:00 +CPU= 85.199
  08:13:57 STEP  598 H= 598:00 +CPU= 53.602
  08:14:50 STEP  599 H= 599:00 +CPU= 53.005
  08:16:36 STEP  600 H= 600:00 +CPU= 99.815
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_24.zip
Uploading the intermediate file: upload_file_24.zip
  08:17:29 STEP  601 H= 601:00 +CPU= 52.266
  08:18:23 STEP  602 H= 602:00 +CPU= 53.726
  08:19:49 STEP  603 H= 603:00 +CPU= 85.209
  08:20:44 STEP  604 H= 604:00 +CPU= 54.008
  08:21:37 STEP  605 H= 605:00 +CPU= 52.354
  08:23:03 STEP  606 H= 606:00 +CPU= 86.200
  08:23:58 STEP  607 H= 607:00 +CPU= 52.486
  08:24:54 STEP  608 H= 608:00 +CPU= 53.681
  08:26:19 STEP  609 H= 609:00 +CPU= 84.453
  08:27:15 STEP  610 H= 610:00 +CPU= 55.173
  08:28:08 STEP  611 H= 611:00 +CPU= 52.511
  08:30:51 STEP  612 H= 612:00 +CPU=109.085
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000612
  08:32:19 STEP  613 H= 613:00 +CPU= 59.241
  08:33:42 STEP  614 H= 614:00 +CPU= 60.053
  08:35:43 STEP  615 H= 615:00 +CPU= 94.186
  08:37:12 STEP  616 H= 616:00 +CPU= 61.211
  08:38:30 STEP  617 H= 617:00 +CPU= 59.119
  08:41:21 STEP  618 H= 618:00 +CPU=103.210
  08:42:57 STEP  619 H= 619:00 +CPU= 61.727
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0780
(argv4) batchid: 989
(argv5) wuid: 12206424
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5e18b625184702a1ee1c0bb61b4c552b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
ic_ancil_file: ic_ancil_12206424
ifsdata_file: ifsdata_12206424
climate_data_file: clim_data_12206424
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9fca43bbc4286d19ab5714e43b24d6a7 to: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bf5d29308356670cc64ac2016e4d4593 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12206424.xml
-- Model is restarting --
Adjusting last_iter, 619, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
last_cpu_time: 42603
upload_file_number: 25
last_iter: 599
last_upload: 2160000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262
The child process has been launched with process id: 531829
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:531829:531829] [20221230:084415:1672411455:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:46:20 STEP  600 H= 600:00 +CPU=101.551
  08:47:41 STEP  601 H= 601:00 +CPU= 59.348
  08:48:40 STEP  602 H= 602:00 +CPU= 54.467
  08:50:48 STEP  603 H= 603:00 +CPU= 94.517
  08:52:04 STEP  604 H= 604:00 +CPU= 59.417
  08:53:11 STEP  605 H= 605:00 +CPU= 55.274
  08:54:39 STEP  606 H= 606:00 +CPU= 86.943
  08:55:32 STEP  607 H= 607:00 +CPU= 52.157
  08:56:26 STEP  608 H= 608:00 +CPU= 53.806
  08:57:53 STEP  609 H= 609:00 +CPU= 86.345
  08:58:47 STEP  610 H= 610:00 +CPU= 53.689
  08:59:41 STEP  611 H= 611:00 +CPU= 52.479
  09:01:18 STEP  612 H= 612:00 +CPU= 95.341
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000612
  09:02:11 STEP  613 H= 613:00 +CPU= 52.732
  09:03:06 STEP  614 H= 614:00 +CPU= 54.471
  09:04:32 STEP  615 H= 615:00 +CPU= 84.931
  09:05:27 STEP  616 H= 616:00 +CPU= 55.234
  09:06:21 STEP  617 H= 617:00 +CPU= 52.646
  09:07:48 STEP  618 H= 618:00 +CPU= 86.885
  09:08:40 STEP  619 H= 619:00 +CPU= 51.984
  09:09:35 STEP  620 H= 620:00 +CPU= 54.022
  09:11:04 STEP  621 H= 621:00 +CPU= 85.340
  09:11:59 STEP  622 H= 622:00 +CPU= 53.945
  09:12:54 STEP  623 H= 623:00 +CPU= 54.744
  09:14:39 STEP  624 H= 624:00 +CPU= 99.282
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_25.zip
Uploading the intermediate file: upload_file_25.zip
  09:15:32 STEP  625 H= 625:00 +CPU= 52.602
  09:16:26 STEP  626 H= 626:00 +CPU= 53.555
  09:17:54 STEP  627 H= 627:00 +CPU= 86.178
  09:18:49 STEP  628 H= 628:00 +CPU= 53.935
  09:19:41 STEP  629 H= 629:00 +CPU= 52.309
  09:21:10 STEP  630 H= 630:00 +CPU= 88.366
  09:22:04 STEP  631 H= 631:00 +CPU= 52.483
  09:22:57 STEP  632 H= 632:00 +CPU= 53.316
  09:24:24 STEP  633 H= 633:00 +CPU= 86.113
  09:25:18 STEP  634 H= 634:00 +CPU= 53.157
  09:26:11 STEP  635 H= 635:00 +CPU= 52.548
  09:27:45 STEP  636 H= 636:00 +CPU= 93.734
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000636
  09:28:39 STEP  637 H= 637:00 +CPU= 52.739
  09:29:33 STEP  638 H= 638:00 +CPU= 54.261
  09:30:58 STEP  639 H= 639:00 +CPU= 83.782
  09:31:51 STEP  640 H= 640:00 +CPU= 52.993
  09:32:44 STEP  641 H= 641:00 +CPU= 51.960
  09:34:11 STEP  642 H= 642:00 +CPU= 86.561
  09:35:03 STEP  643 H= 643:00 +CPU= 51.522
  09:35:56 STEP  644 H= 644:00 +CPU= 52.860
  09:37:22 STEP  645 H= 645:00 +CPU= 84.643
  09:38:17 STEP  646 H= 646:00 +CPU= 54.417
  09:39:11 STEP  647 H= 647:00 +CPU= 53.537
  09:40:53 STEP  648 H= 648:00 +CPU= 99.239
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_26.zip
Uploading the intermediate file: upload_file_26.zip
  09:41:46 STEP  649 H= 649:00 +CPU= 52.174
  09:42:40 STEP  650 H= 650:00 +CPU= 54.020
  09:44:07 STEP  651 H= 651:00 +CPU= 84.774
  09:45:06 STEP  652 H= 652:00 +CPU= 54.574
  09:46:13 STEP  653 H= 653:00 +CPU= 54.779
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0780
(argv4) batchid: 989
(argv5) wuid: 12206424
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5e18b625184702a1ee1c0bb61b4c552b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
ic_ancil_file: ic_ancil_12206424
ifsdata_file: ifsdata_12206424
climate_data_file: clim_data_12206424
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9fca43bbc4286d19ab5714e43b24d6a7 to: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bf5d29308356670cc64ac2016e4d4593 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12206424.xml
-- Model is restarting --
Adjusting last_iter, 654, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
last_cpu_time: 46192
upload_file_number: 27
last_iter: 647
last_upload: 2332800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262
The child process has been launched with process id: 532371
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:532371:532371] [20221230:095406:1672415646:0.002] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:56:20 STEP  648 H= 648:00 +CPU=103.500
  09:57:59 STEP  649 H= 649:00 +CPU= 61.729
  09:59:29 STEP  650 H= 650:00 +CPU= 61.686
  10:01:42 STEP  651 H= 651:00 +CPU= 97.444
  10:03:04 STEP  652 H= 652:00 +CPU= 60.635
  10:04:35 STEP  653 H= 653:00 +CPU= 59.942
  10:07:07 STEP  654 H= 654:00 +CPU=101.415
  10:08:19 STEP  655 H= 655:00 +CPU= 57.101
  10:09:22 STEP  656 H= 656:00 +CPU= 56.141
  10:11:33 STEP  657 H= 657:00 +CPU= 95.892
  10:12:33 STEP  658 H= 658:00 +CPU= 55.217
  10:13:54 STEP  659 H= 659:00 +CPU= 58.885
  10:15:49 STEP  660 H= 660:00 +CPU= 99.239
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000660
  10:17:10 STEP  661 H= 661:00 +CPU= 59.047
  10:18:32 STEP  662 H= 662:00 +CPU= 60.158
  10:20:12 STEP  663 H= 663:00 +CPU= 87.969
  10:21:05 STEP  664 H= 664:00 +CPU= 53.489
  10:21:59 STEP  665 H= 665:00 +CPU= 52.929
  10:23:28 STEP  666 H= 666:00 +CPU= 88.078
  10:24:20 STEP  667 H= 667:00 +CPU= 51.838
  10:25:14 STEP  668 H= 668:00 +CPU= 53.644
  10:26:40 STEP  669 H= 669:00 +CPU= 85.819
  10:27:35 STEP  670 H= 670:00 +CPU= 53.884
  10:28:30 STEP  671 H= 671:00 +CPU= 54.553
  10:30:13 STEP  672 H= 672:00 +CPU=100.989
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_27.zip
Uploading the intermediate file: upload_file_27.zip
  10:31:05 STEP  673 H= 673:00 +CPU= 51.305
  10:31:59 STEP  674 H= 674:00 +CPU= 53.253
  10:33:26 STEP  675 H= 675:00 +CPU= 85.836
  10:34:20 STEP  676 H= 676:00 +CPU= 53.689
  10:35:12 STEP  677 H= 677:00 +CPU= 51.915
  10:36:40 STEP  678 H= 678:00 +CPU= 87.034
  10:37:33 STEP  679 H= 679:00 +CPU= 52.156
  10:38:27 STEP  680 H= 680:00 +CPU= 53.358
  10:39:53 STEP  681 H= 681:00 +CPU= 85.551
  10:40:48 STEP  682 H= 682:00 +CPU= 54.272
  10:41:40 STEP  683 H= 683:00 +CPU= 51.892
  10:43:18 STEP  684 H= 684:00 +CPU= 96.400
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000684
  10:44:12 STEP  685 H= 685:00 +CPU= 53.080
  10:45:06 STEP  686 H= 686:00 +CPU= 53.850
  10:46:33 STEP  687 H= 687:00 +CPU= 85.923
  10:47:27 STEP  688 H= 688:00 +CPU= 53.655
  10:48:20 STEP  689 H= 689:00 +CPU= 52.652
  10:49:48 STEP  690 H= 690:00 +CPU= 87.205
  10:50:40 STEP  691 H= 691:00 +CPU= 51.905
  10:51:34 STEP  692 H= 692:00 +CPU= 53.200
  10:53:00 STEP  693 H= 693:00 +CPU= 84.772
  10:53:54 STEP  694 H= 694:00 +CPU= 54.110
  10:54:49 STEP  695 H= 695:00 +CPU= 54.304
  10:56:31 STEP  696 H= 696:00 +CPU= 98.216
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_28.zip
Uploading the intermediate file: upload_file_28.zip
  10:57:25 STEP  697 H= 697:00 +CPU= 52.806
  10:58:19 STEP  698 H= 698:00 +CPU= 53.853
  10:59:47 STEP  699 H= 699:00 +CPU= 85.862
  11:00:42 STEP  700 H= 700:00 +CPU= 54.662
  11:01:36 STEP  701 H= 701:00 +CPU= 53.082
  11:03:04 STEP  702 H= 702:00 +CPU= 87.733
  11:03:57 STEP  703 H= 703:00 +CPU= 52.026
  11:04:52 STEP  704 H= 704:00 +CPU= 54.739
  11:06:20 STEP  705 H= 705:00 +CPU= 86.868
  11:07:15 STEP  706 H= 706:00 +CPU= 54.438
  11:08:08 STEP  707 H= 707:00 +CPU= 52.234
  11:09:43 STEP  708 H= 708:00 +CPU= 94.419
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000708
  11:10:36 STEP  709 H= 709:00 +CPU= 52.436
  11:11:30 STEP  710 H= 710:00 +CPU= 53.174
  11:12:58 STEP  711 H= 711:00 +CPU= 87.166
  11:13:52 STEP  712 H= 712:00 +CPU= 54.378
  11:14:44 STEP  713 H= 713:00 +CPU= 51.538
  11:16:12 STEP  714 H= 714:00 +CPU= 87.423
  11:17:05 STEP  715 H= 715:00 +CPU= 52.226
  11:17:58 STEP  716 H= 716:00 +CPU= 52.951
  11:19:23 STEP  717 H= 717:00 +CPU= 84.187
  11:20:28 STEP  718 H= 718:00 +CPU= 56.146
  11:21:42 STEP  719 H= 719:00 +CPU= 58.906
  11:24:40 STEP  720 H= 720:00 +CPU=114.378
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_29.zip
Uploading the intermediate file: upload_file_29.zip
  11:26:09 STEP  721 H= 721:00 +CPU= 61.325
  11:27:15 STEP  722 H= 722:00 +CPU= 57.552
  11:29:33 STEP  723 H= 723:00 +CPU= 96.235
  11:31:09 STEP  724 H= 724:00 +CPU= 63.455
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0780
(argv4) batchid: 989
(argv5) wuid: 12206424
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5e18b625184702a1ee1c0bb61b4c552b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
ic_ancil_file: ic_ancil_12206424
ifsdata_file: ifsdata_12206424
climate_data_file: clim_data_12206424
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9fca43bbc4286d19ab5714e43b24d6a7 to: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bf5d29308356670cc64ac2016e4d4593 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12206424.xml
-- Model is restarting --
Adjusting last_iter, 725, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
last_cpu_time: 51404
upload_file_number: 30
last_iter: 719
last_upload: 2592000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262
The child process has been launched with process id: 533159
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533159:533159] [20221230:113234:1672421554:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:34:39 STEP  720 H= 720:00 +CPU=102.359
  11:35:59 STEP  721 H= 721:00 +CPU= 58.764
  11:37:00 STEP  722 H= 722:00 +CPU= 55.404
  11:39:23 STEP  723 H= 723:00 +CPU= 97.055
  11:40:47 STEP  724 H= 724:00 +CPU= 59.708
  11:41:50 STEP  725 H= 725:00 +CPU= 56.481
  11:44:15 STEP  726 H= 726:00 +CPU= 98.275
  11:45:43 STEP  727 H= 727:00 +CPU= 60.693
  11:46:40 STEP  728 H= 728:00 +CPU= 54.593
  11:48:17 STEP  729 H= 729:00 +CPU= 88.700
  11:49:11 STEP  730 H= 730:00 +CPU= 53.426
  11:50:05 STEP  731 H= 731:00 +CPU= 53.681
  11:51:42 STEP  732 H= 732:00 +CPU= 96.324
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000732
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000732
  11:52:34 STEP  733 H= 733:00 +CPU= 51.225
  11:53:29 STEP  734 H= 734:00 +CPU= 53.862
  11:54:55 STEP  735 H= 735:00 +CPU= 85.780
  11:55:49 STEP  736 H= 736:00 +CPU= 53.367
  11:56:42 STEP  737 H= 737:00 +CPU= 52.301
  11:58:10 STEP  738 H= 738:00 +CPU= 87.103
  11:59:03 STEP  739 H= 739:00 +CPU= 51.549
  11:59:58 STEP  740 H= 740:00 +CPU= 53.683
  12:01:25 STEP  741 H= 741:00 +CPU= 85.543
  12:02:20 STEP  742 H= 742:00 +CPU= 53.852
  12:03:14 STEP  743 H= 743:00 +CPU= 53.738
  12:04:55 STEP  744 H= 744:00 +CPU= 98.264
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000744
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000732
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000732
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_30.zip
Uploading the intermediate file: upload_file_30.zip
  12:05:48 STEP  745 H= 745:00 +CPU= 52.103
  12:06:42 STEP  746 H= 746:00 +CPU= 53.810
  12:08:10 STEP  747 H= 747:00 +CPU= 85.977
  12:09:05 STEP  748 H= 748:00 +CPU= 53.606
  12:09:58 STEP  749 H= 749:00 +CPU= 51.988
  12:11:27 STEP  750 H= 750:00 +CPU= 87.033
  12:12:21 STEP  751 H= 751:00 +CPU= 52.451
  12:13:14 STEP  752 H= 752:00 +CPU= 52.794
  12:14:42 STEP  753 H= 753:00 +CPU= 85.779
  12:15:37 STEP  754 H= 754:00 +CPU= 54.076
  12:16:30 STEP  755 H= 755:00 +CPU= 52.088
  12:18:06 STEP  756 H= 756:00 +CPU= 95.101
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000756
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000756
  12:18:59 STEP  757 H= 757:00 +CPU= 52.194
  12:19:55 STEP  758 H= 758:00 +CPU= 54.980
  12:21:24 STEP  759 H= 759:00 +CPU= 86.747
  12:22:18 STEP  760 H= 760:00 +CPU= 53.431
  12:23:11 STEP  761 H= 761:00 +CPU= 52.616
  12:24:39 STEP  762 H= 762:00 +CPU= 87.383
  12:25:32 STEP  763 H= 763:00 +CPU= 51.768
  12:26:27 STEP  764 H= 764:00 +CPU= 54.241
  12:27:52 STEP  765 H= 765:00 +CPU= 84.383
  12:28:45 STEP  766 H= 766:00 +CPU= 53.619
  12:29:40 STEP  767 H= 767:00 +CPU= 53.409
  12:31:23 STEP  768 H= 768:00 +CPU= 98.952
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000768
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000744
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000756
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000756
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_31.zip
Uploading the intermediate file: upload_file_31.zip
  12:32:17 STEP  769 H= 769:00 +CPU= 53.013
  12:33:10 STEP  770 H= 770:00 +CPU= 52.917
  12:34:40 STEP  771 H= 771:00 +CPU= 85.631
  12:35:34 STEP  772 H= 772:00 +CPU= 53.085
  12:36:27 STEP  773 H= 773:00 +CPU= 52.523
  12:37:54 STEP  774 H= 774:00 +CPU= 85.909
  12:38:47 STEP  775 H= 775:00 +CPU= 52.435
  12:39:41 STEP  776 H= 776:00 +CPU= 53.648
  12:41:06 STEP  777 H= 777:00 +CPU= 84.465
  12:42:01 STEP  778 H= 778:00 +CPU= 54.062
  12:42:54 STEP  779 H= 779:00 +CPU= 52.534
  12:44:29 STEP  780 H= 780:00 +CPU= 94.569
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780
  12:45:25 STEP  781 H= 781:00 +CPU= 53.353
  12:46:33 STEP  782 H= 782:00 +CPU= 56.443
  12:48:26 STEP  783 H= 783:00 +CPU= 91.636
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2020050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0780
(argv4) batchid: 989
(argv5) wuid: 12206424
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5e18b625184702a1ee1c0bb61b4c552b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0780_2020050100_123_989_12206424.zip
ic_ancil_file: ic_ancil_12206424
ifsdata_file: ifsdata_12206424
climate_data_file: clim_data_12206424
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9fca43bbc4286d19ab5714e43b24d6a7 to: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12206424.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bf5d29308356670cc64ac2016e4d4593 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12206424.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12206424.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12206424.xml
-- Model is restarting --
Adjusting last_iter, 784, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12206424.xml
last_cpu_time: 55688
upload_file_number: 32
last_iter: 767
last_upload: 2764800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262
The child process has been launched with process id: 533829
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:dmz1:1:1:533829:533829] [20221230:125022:1672426222:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  12:52:23 STEP  768 H= 768:00 +CPU=100.630
  12:53:49 STEP  769 H= 769:00 +CPU= 58.916
  12:55:16 STEP  770 H= 770:00 +CPU= 62.718
  12:57:16 STEP  771 H= 771:00 +CPU= 91.930
  12:58:12 STEP  772 H= 772:00 +CPU= 54.404
  12:59:05 STEP  773 H= 773:00 +CPU= 52.268
  13:00:34 STEP  774 H= 774:00 +CPU= 87.424
  13:01:27 STEP  775 H= 775:00 +CPU= 52.783
  13:02:21 STEP  776 H= 776:00 +CPU= 53.762
  13:03:46 STEP  777 H= 777:00 +CPU= 83.630
  13:04:39 STEP  778 H= 778:00 +CPU= 53.521
  13:05:34 STEP  779 H= 779:00 +CPU= 53.433
  13:07:09 STEP  780 H= 780:00 +CPU= 94.708
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000780
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000780
  13:08:03 STEP  781 H= 781:00 +CPU= 53.617
  13:08:59 STEP  782 H= 782:00 +CPU= 54.684
  13:10:30 STEP  783 H= 783:00 +CPU= 87.973
  13:11:24 STEP  784 H= 784:00 +CPU= 53.658
  13:12:26 STEP  785 H= 785:00 +CPU= 53.995
  13:13:54 STEP  786 H= 786:00 +CPU= 87.480
  13:14:46 STEP  787 H= 787:00 +CPU= 52.059
  13:15:44 STEP  788 H= 788:00 +CPU= 54.213
  13:17:19 STEP  789 H= 789:00 +CPU= 87.330
  13:18:13 STEP  790 H= 790:00 +CPU= 53.090
  13:19:07 STEP  791 H= 791:00 +CPU= 53.706
  13:20:57 STEP  792 H= 792:00 +CPU=101.245
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000792
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000792
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000768
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMGGhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMSHhq0f+000780
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12206424/ICMUAhq0f+000780
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0780_2020050100_123_989_12206424_1_r851479262_32.zip
Uploading the intermediate file: upload_file_32.zip
  13:21:51 STEP  793 H= 793:00 +CPU= 52.339
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:02:34 1458811 22294042 oifs_43r3_ps_0780_2020050100_123_989_12206424_1 864,000 15,379 0.0178


©2024 cpdn.org