climateprediction.net home page
Task 22270207

Task 22270207

Name oifs_43r3_ps_0286_1999050100_123_968_12184930_0
Workunit 12184930
Created 22 Dec 2022, 18:25:46 UTC
Sent 17 Jan 2023, 21:32:05 UTC
Report deadline 18 Mar 2023, 21:32:05 UTC
Received 20 Jan 2023, 17:04:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 8 hours 21 min 28 sec
CPU time 5 hours 49 min 33 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,418.04 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,676.57 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
STEP  530 H= 530:00 +CPU= 21.349
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:11:07 STEP  531 H= 531:00 +CPU= 32.871
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0286
(argv4) batchid: 968
(argv5) wuid: 12184930
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_41980b02c092bc78b64afd212def4d8f to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0286_1999050100_123_968_12184930.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0286_1999050100_123_968_12184930.zip
ic_ancil_file: ic_ancil_12184930
ifsdata_file: ifsdata_12184930
climate_data_file: clim_data_12184930
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_add0d6b2d2d22ca2a8c217229ba33bff to: /var/lib/boinc-client/slots/11/ic_ancil_12184930.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12184930.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_262f5063fb8838d221c7d356f1d0579c to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12184930.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12184930.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12184930.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12184930.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12184930.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12184930.xml
-- Model is restarting --
Adjusting last_iter, 532, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12184930.xml
last_cpu_time: 14969
upload_file_number: 22
last_iter: 527
last_upload: 1900800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620
The child process has been launched with process id: 216251
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:216251:216251] [20230120:111410:1674180850:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:14:49 STEP  528 H= 528:00 +CPU= 36.752
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:16:51 STEP  529 H= 529:00 +CPU= 20.507
  11:17:12 STEP  530 H= 530:00 +CPU= 21.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:19:56 STEP  531 H= 531:00 +CPU= 32.702
  11:20:17 STEP  532 H= 532:00 +CPU= 21.074
  11:20:37 STEP  533 H= 533:00 +CPU= 20.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:23:22 STEP  534 H= 534:00 +CPU= 33.642
  11:23:43 STEP  535 H= 535:00 +CPU= 20.263
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:25:55 STEP  536 H= 536:00 +CPU= 21.253
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:28:49 STEP  537 H= 537:00 +CPU= 32.809
  11:29:11 STEP  538 H= 538:00 +CPU= 21.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:31:03 STEP  539 H= 539:00 +CPU= 20.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:33:31 STEP  540 H= 540:00 +CPU= 36.112
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000540
  11:33:52 STEP  541 H= 541:00 +CPU= 20.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:55 STEP  542 H= 542:00 +CPU= 21.343
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:38:19 STEP  543 H= 543:00 +CPU= 33.235
  11:38:40 STEP  544 H= 544:00 +CPU= 21.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:41:02 STEP  545 H= 545:00 +CPU= 20.662
  11:41:36 STEP  546 H= 546:00 +CPU= 33.325
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:43:58 STEP  547 H= 547:00 +CPU= 20.753
  11:44:19 STEP  548 H= 548:00 +CPU= 21.094
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:33 STEP  549 H= 549:00 +CPU= 33.109
  11:46:55 STEP  550 H= 550:00 +CPU= 21.429
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:17 STEP  551 H= 551:00 +CPU= 21.138
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:51:57 STEP  552 H= 552:00 +CPU= 38.670
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_22.zip
Uploading the intermediate file: upload_file_22.zip
  11:52:17 STEP  553 H= 553:00 +CPU= 19.975
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:54:29 STEP  554 H= 554:00 +CPU= 20.885
  11:55:02 STEP  555 H= 555:00 +CPU= 32.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:34 STEP  556 H= 556:00 +CPU= 20.893
  11:57:54 STEP  557 H= 557:00 +CPU= 20.209
  11:58:27 STEP  558 H= 558:00 +CPU= 32.947
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:29 STEP  559 H= 559:00 +CPU= 20.377
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:03:41 STEP  560 H= 560:00 +CPU= 21.180
  12:04:14 STEP  561 H= 561:00 +CPU= 32.712
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:06:27 STEP  562 H= 562:00 +CPU= 21.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:48 STEP  563 H= 563:00 +CPU= 20.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:05 STEP  564 H= 564:00 +CPU= 35.906
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000564
  12:10:27 STEP  565 H= 565:00 +CPU= 21.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:50 STEP  566 H= 566:00 +CPU= 21.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:15:24 STEP  567 H= 567:00 +CPU= 32.872
  12:15:44 STEP  568 H= 568:00 +CPU= 20.311
  12:16:04 STEP  569 H= 569:00 +CPU= 20.130
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:18:39 STEP  570 H= 570:00 +CPU= 33.402
  12:18:59 STEP  571 H= 571:00 +CPU= 20.136
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:21:23 STEP  572 H= 572:00 +CPU= 21.838
  12:21:56 STEP  573 H= 573:00 +CPU= 32.055
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:24:18 STEP  574 H= 574:00 +CPU= 21.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:26:31 STEP  575 H= 575:00 +CPU= 21.452
  12:27:09 STEP  576 H= 576:00 +CPU= 38.078
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_23.zip
Uploading the intermediate file: upload_file_23.zip
  12:27:29 STEP  577 H= 577:00 +CPU= 19.660
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:30:00 STEP  578 H= 578:00 +CPU= 20.549
  12:30:32 STEP  579 H= 579:00 +CPU= 32.029
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:33:26 STEP  580 H= 580:00 +CPU= 21.721
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:35:18 STEP  581 H= 581:00 +CPU= 20.267
  12:35:52 STEP  582 H= 582:00 +CPU= 33.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:38:24 STEP  583 H= 583:00 +CPU= 20.747
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:40:37 STEP  584 H= 584:00 +CPU= 20.668
  12:41:10 STEP  585 H= 585:00 +CPU= 32.991
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:43:52 STEP  586 H= 586:00 +CPU= 20.910
  12:44:13 STEP  587 H= 587:00 +CPU= 20.572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:47:01 STEP  588 H= 588:00 +CPU= 36.736
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000588
  12:47:22 STEP  589 H= 589:00 +CPU= 20.481
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:49:43 STEP  590 H= 590:00 +CPU= 20.725
  12:50:16 STEP  591 H= 591:00 +CPU= 32.771
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:53:50 STEP  592 H= 592:00 +CPU= 21.166
  12:54:11 STEP  593 H= 593:00 +CPU= 21.093
  12:54:47 STEP  594 H= 594:00 +CPU= 35.150
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:00:22 STEP  595 H= 595:00 +CPU= 22.526
  13:00:45 STEP  596 H= 596:00 +CPU= 22.627
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:08:21 STEP  597 H= 597:00 +CPU= 36.232
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:21:58 STEP  598 H= 598:00 +CPU= 22.692
  20:22:20 STEP  599 H= 599:00 +CPU= 22.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:31:46 STEP  600 H= 600:00 +CPU= 42.224
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_24.zip
Uploading the intermediate file: upload_file_24.zip
  20:32:09 STEP  601 H= 601:00 +CPU= 23.133
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:36:55 STEP  602 H= 602:00 +CPU= 23.775
  20:37:30 STEP  603 H= 603:00 +CPU= 34.275
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:43:35 STEP  604 H= 604:00 +CPU= 22.650
  20:43:57 STEP  605 H= 605:00 +CPU= 21.929
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:49:25 STEP  606 H= 606:00 +CPU= 35.587
  20:49:46 STEP  607 H= 607:00 +CPU= 21.305
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:52:50 STEP  608 H= 608:00 +CPU= 21.803
  20:53:25 STEP  609 H= 609:00 +CPU= 34.368
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:59:31 STEP  610 H= 610:00 +CPU= 23.764
  20:59:54 STEP  611 H= 611:00 +CPU= 23.152
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:07:19 STEP  612 H= 612:00 +CPU= 41.108
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000612
  21:07:43 STEP  613 H= 613:00 +CPU= 23.317
  21:08:07 STEP  614 H= 614:00 +CPU= 23.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:14:17 STEP  615 H= 615:00 +CPU= 37.615
  21:14:41 STEP  616 H= 616:00 +CPU= 24.112
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:19:47 STEP  617 H= 617:00 +CPU= 23.010
  21:20:25 STEP  618 H= 618:00 +CPU= 37.651
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:25:11 STEP  619 H= 619:00 +CPU= 23.183
  21:25:36 STEP  620 H= 620:00 +CPU= 24.130
  21:26:15 STEP  621 H= 621:00 +CPU= 37.650
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:31:21 STEP  622 H= 622:00 +CPU= 23.640
  21:31:45 STEP  623 H= 623:00 +CPU= 23.652
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:37:42 STEP  624 H= 624:00 +CPU= 43.242
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_25.zip
Uploading the intermediate file: upload_file_25.zip
  21:38:05 STEP  625 H= 625:00 +CPU= 23.345
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:42:30 STEP  626 H= 626:00 +CPU= 23.995
  21:43:07 STEP  627 H= 627:00 +CPU= 36.150
  21:43:31 STEP  628 H= 628:00 +CPU= 23.765
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:49:37 STEP  629 H= 629:00 +CPU= 23.472
  21:50:15 STEP  630 H= 630:00 +CPU= 37.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:55:01 STEP  631 H= 631:00 +CPU= 23.406
  21:55:26 STEP  632 H= 632:00 +CPU= 23.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:01:05 STEP  633 H= 633:00 +CPU= 36.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:05:01 STEP  634 H= 634:00 +CPU= 24.121
  22:05:25 STEP  635 H= 635:00 +CPU= 23.461
  22:06:06 STEP  636 H= 636:00 +CPU= 40.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000636
  22:10:51 STEP  637 H= 637:00 +CPU= 23.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:14:27 STEP  638 H= 638:00 +CPU= 23.645
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:18:06 STEP  639 H= 639:00 +CPU= 37.044
  22:18:31 STEP  640 H= 640:00 +CPU= 24.237
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0286
(argv4) batchid: 968
(argv5) wuid: 12184930
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_41980b02c092bc78b64afd212def4d8f to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0286_1999050100_123_968_12184930.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0286_1999050100_123_968_12184930.zip
ic_ancil_file: ic_ancil_12184930
ifsdata_file: ifsdata_12184930
climate_data_file: clim_data_12184930
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_add0d6b2d2d22ca2a8c217229ba33bff to: /var/lib/boinc-client/slots/11/ic_ancil_12184930.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12184930.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_262f5063fb8838d221c7d356f1d0579c to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12184930.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12184930.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12184930.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12184930.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12184930.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12184930.xml
-- Model is restarting --
Adjusting last_iter, 640, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12184930.xml
last_cpu_time: 17959
upload_file_number: 26
last_iter: 623
last_upload: 2246400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620
The child process has been launched with process id: 333888
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:333888:333888] [20230120:222331:1674221011:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:24:15 STEP  624 H= 624:00 +CPU= 40.904
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:50 STEP  625 H= 625:00 +CPU= 22.763
  22:29:14 STEP  626 H= 626:00 +CPU= 23.615
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:35:04 STEP  627 H= 627:00 +CPU= 37.519
  22:35:28 STEP  628 H= 628:00 +CPU= 23.739
  22:35:51 STEP  629 H= 629:00 +CPU= 22.959
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:51:04 STEP  630 H= 630:00 +CPU= 37.329
  22:51:25 STEP  631 H= 631:00 +CPU= 20.423
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:06 STEP  632 H= 632:00 +CPU= 21.187
  22:53:38 STEP  633 H= 633:00 +CPU= 31.998
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:55:20 STEP  634 H= 634:00 +CPU= 20.922
  22:55:40 STEP  635 H= 635:00 +CPU= 19.918
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:48 STEP  636 H= 636:00 +CPU= 35.618
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000636
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000636
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:59:18 STEP  637 H= 637:00 +CPU= 20.130
  22:59:39 STEP  638 H= 638:00 +CPU= 20.778
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:02:03 STEP  639 H= 639:00 +CPU= 31.968
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:03:25 STEP  640 H= 640:00 +CPU= 20.551
  23:03:45 STEP  641 H= 641:00 +CPU= 19.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:05:09 STEP  642 H= 642:00 +CPU= 31.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:06:40 STEP  643 H= 643:00 +CPU= 20.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:08:04 STEP  644 H= 644:00 +CPU= 21.533
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:09:48 STEP  645 H= 645:00 +CPU= 31.891
  23:10:09 STEP  646 H= 646:00 +CPU= 20.728
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:11:29 STEP  647 H= 647:00 +CPU= 20.508
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:13:06 STEP  648 H= 648:00 +CPU= 36.659
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000648
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000636
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000636
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_26.zip
Uploading the intermediate file: upload_file_26.zip
  23:13:26 STEP  649 H= 649:00 +CPU= 19.534
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:14:45 STEP  650 H= 650:00 +CPU= 20.036
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:16:48 STEP  651 H= 651:00 +CPU= 31.838
  23:17:09 STEP  652 H= 652:00 +CPU= 20.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:18:40 STEP  653 H= 653:00 +CPU= 19.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:20:22 STEP  654 H= 654:00 +CPU= 32.566
  23:20:42 STEP  655 H= 655:00 +CPU= 20.033
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:22:03 STEP  656 H= 656:00 +CPU= 20.732
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:23:38 STEP  657 H= 657:00 +CPU= 32.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:19 STEP  658 H= 658:00 +CPU= 20.627
  23:25:40 STEP  659 H= 659:00 +CPU= 20.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:27:35 STEP  660 H= 660:00 +CPU= 34.713
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000660
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000660
  23:27:55 STEP  661 H= 661:00 +CPU= 20.399
  23:28:16 STEP  662 H= 662:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:30:19 STEP  663 H= 663:00 +CPU= 31.983
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:31:41 STEP  664 H= 664:00 +CPU= 20.923
  23:32:01 STEP  665 H= 665:00 +CPU= 19.980
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:33:46 STEP  666 H= 666:00 +CPU= 32.469
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:35:17 STEP  667 H= 667:00 +CPU= 19.902
  23:35:37 STEP  668 H= 668:00 +CPU= 20.866
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:37:31 STEP  669 H= 669:00 +CPU= 31.687
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:38:42 STEP  670 H= 670:00 +CPU= 20.291
  23:39:02 STEP  671 H= 671:00 +CPU= 20.565
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:41:21 STEP  672 H= 672:00 +CPU= 36.853
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000672
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000648
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000660
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000660
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_27.zip
Uploading the intermediate file: upload_file_27.zip
  23:41:40 STEP  673 H= 673:00 +CPU= 19.079
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:43:06 STEP  674 H= 674:00 +CPU= 19.387
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:44:49 STEP  675 H= 675:00 +CPU= 31.756
  23:45:09 STEP  676 H= 676:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:46:51 STEP  677 H= 677:00 +CPU= 20.111
  23:47:23 STEP  678 H= 678:00 +CPU= 32.698
  23:47:43 STEP  679 H= 679:00 +CPU= 19.497
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:49:26 STEP  680 H= 680:00 +CPU= 20.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:51:09 STEP  681 H= 681:00 +CPU= 32.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:31 STEP  682 H= 682:00 +CPU= 20.457
  23:52:51 STEP  683 H= 683:00 +CPU= 20.144
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:48 STEP  684 H= 684:00 +CPU= 35.223
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000684
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000684
  23:55:07 STEP  685 H= 685:00 +CPU= 19.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:56:40 STEP  686 H= 686:00 +CPU= 21.141
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:58:23 STEP  687 H= 687:00 +CPU= 31.917
  23:58:44 STEP  688 H= 688:00 +CPU= 20.542
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:00:05 STEP  689 H= 689:00 +CPU= 19.872
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:01:48 STEP  690 H= 690:00 +CPU= 32.706
  00:02:09 STEP  691 H= 691:00 +CPU= 20.788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:03:21 STEP  692 H= 692:00 +CPU= 20.449
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:05:05 STEP  693 H= 693:00 +CPU= 31.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:06:37 STEP  694 H= 694:00 +CPU= 20.918
  00:06:58 STEP  695 H= 695:00 +CPU= 20.525
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:09:18 STEP  696 H= 696:00 +CPU= 37.693
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000696
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000672
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000684
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_28.zip
Uploading the intermediate file: upload_file_28.zip
  00:09:37 STEP  697 H= 697:00 +CPU= 19.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:11:29 STEP  698 H= 698:00 +CPU= 21.400
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:13:34 STEP  699 H= 699:00 +CPU= 32.551
  00:13:54 STEP  700 H= 700:00 +CPU= 20.618
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:15:25 STEP  701 H= 701:00 +CPU= 19.737
  00:15:58 STEP  702 H= 702:00 +CPU= 32.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:17:38 STEP  703 H= 703:00 +CPU= 20.014
  00:17:59 STEP  704 H= 704:00 +CPU= 21.023
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:19:53 STEP  705 H= 705:00 +CPU= 32.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:21:16 STEP  706 H= 706:00 +CPU= 21.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:22:19 STEP  707 H= 707:00 +CPU= 21.116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:23:45 STEP  708 H= 708:00 +CPU= 35.261
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000708
  00:24:05 STEP  709 H= 709:00 +CPU= 20.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:25:47 STEP  710 H= 710:00 +CPU= 20.952
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:27:40 STEP  711 H= 711:00 +CPU= 32.127
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1999050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0286
(argv4) batchid: 968
(argv5) wuid: 12184930
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_41980b02c092bc78b64afd212def4d8f to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0286_1999050100_123_968_12184930.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0286_1999050100_123_968_12184930.zip
ic_ancil_file: ic_ancil_12184930
ifsdata_file: ifsdata_12184930
climate_data_file: clim_data_12184930
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_add0d6b2d2d22ca2a8c217229ba33bff to: /var/lib/boinc-client/slots/11/ic_ancil_12184930.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12184930.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_262f5063fb8838d221c7d356f1d0579c to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12184930.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12184930.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12184930.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12184930.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12184930.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12184930.xml
-- Model is restarting --
Adjusting last_iter, 711, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12184930.xml
last_cpu_time: 20163
upload_file_number: 29
last_iter: 695
last_upload: 2505600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620
The child process has been launched with process id: 352279
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:352279:352279] [20230121:003014:1674228614:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:30:52 STEP  696 H= 696:00 +CPU= 35.492
  00:31:11 STEP  697 H= 697:00 +CPU= 19.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:33:03 STEP  698 H= 698:00 +CPU= 20.727
  00:33:34 STEP  699 H= 699:00 +CPU= 30.962
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:35:05 STEP  700 H= 700:00 +CPU= 20.433
  00:35:24 STEP  701 H= 701:00 +CPU= 19.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:37:08 STEP  702 H= 702:00 +CPU= 32.180
  00:37:27 STEP  703 H= 703:00 +CPU= 19.345
  00:37:48 STEP  704 H= 704:00 +CPU= 20.556
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:39:51 STEP  705 H= 705:00 +CPU= 31.701
  00:40:10 STEP  706 H= 706:00 +CPU= 19.788
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:41:40 STEP  707 H= 707:00 +CPU= 19.851
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:43:27 STEP  708 H= 708:00 +CPU= 35.116
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000708
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000708
  00:43:47 STEP  709 H= 709:00 +CPU= 20.705
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:45:29 STEP  710 H= 710:00 +CPU= 20.632
  00:46:01 STEP  711 H= 711:00 +CPU= 31.949
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:47:34 STEP  712 H= 712:00 +CPU= 20.958
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:48:57 STEP  713 H= 713:00 +CPU= 20.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:50:42 STEP  714 H= 714:00 +CPU= 33.880
  00:51:02 STEP  715 H= 715:00 +CPU= 20.195
  00:51:22 STEP  716 H= 716:00 +CPU= 20.529
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:53:15 STEP  717 H= 717:00 +CPU= 31.940
  00:53:36 STEP  718 H= 718:00 +CPU= 20.534
  00:53:56 STEP  719 H= 719:00 +CPU= 20.396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:56:15 STEP  720 H= 720:00 +CPU= 37.625
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+000720
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+000720
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000696
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMGGhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMSHhq0f+000708
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12184930/ICMUAhq0f+000708
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0286_1999050100_123_968_12184930_0_r930527620_29.zip
Uploading the intermediate file: upload_file_29.zip
  00:56:35 STEP  721 H= 721:00 +CPU= 19.960
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:58:27 STEP  722 H= 722:00 +CPU= 20.616
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:59:50 STEP  723 H= 723:00 +CPU= 32.313
  01:00:11 STEP  724 H= 724:00 +CPU= 20.200
  01:00:31 STEP  725 H= 725:00 +CPU= 19.827
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:02:26 STEP  726 H= 726:00 +CPU= 33.134
  01:02:46 STEP  727 H= 727:00 +CPU= 19.740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net