climateprediction.net home page
Task 22295338

Task 22295338

Name oifs_43r3_ps_0681_2012050100_123_981_12198325_1
Workunit 12198325
Created 3 Jan 2023, 17:37:24 UTC
Sent 17 Jan 2023, 21:32:05 UTC
Report deadline 18 Mar 2023, 21:32:05 UTC
Received 19 Jan 2023, 15:55:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 5 hours 11 min 47 sec
CPU time 5 hours 3 min 20 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.98 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,688.40 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
hild process
Resuming the child process
  13:45:51 STEP  453 H= 453:00 +CPU= 32.504
  13:46:13 STEP  454 H= 454:00 +CPU= 21.293
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:47:54 STEP  455 H= 455:00 +CPU= 21.306
  13:48:32 STEP  456 H= 456:00 +CPU= 37.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_18.zip
Uploading the intermediate file: upload_file_18.zip
  13:50:03 STEP  457 H= 457:00 +CPU= 21.001
  13:50:25 STEP  458 H= 458:00 +CPU= 21.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:52:29 STEP  459 H= 459:00 +CPU= 33.001
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:54:12 STEP  460 H= 460:00 +CPU= 21.592
  13:54:32 STEP  461 H= 461:00 +CPU= 20.182
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:57:28 STEP  462 H= 462:00 +CPU= 33.743
  13:57:49 STEP  463 H= 463:00 +CPU= 20.960
  13:58:10 STEP  464 H= 464:00 +CPU= 21.311
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:00:04 STEP  465 H= 465:00 +CPU= 32.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:01:27 STEP  466 H= 466:00 +CPU= 21.511
  14:01:47 STEP  467 H= 467:00 +CPU= 20.037
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:03:34 STEP  468 H= 468:00 +CPU= 36.202
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000468
  14:03:55 STEP  469 H= 469:00 +CPU= 20.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3547MB (heap), 4418MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3547 MB (maxheap), 4418 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.149] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.150] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CPG_DRV 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.151] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-CPG                  (1025) 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.151] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              CPG 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.151] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CPG_GP 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153853:1674110333:8366.151] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                GPGRP 
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153854:1674110334:8367.151] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:p500:1:1:3285634:3285634] [20230119:153854:1674110334:8367.151] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000C72D73  gpgrp_                    270  gpgrp.F90
oifs_43r3_model.e  0000000000C3A544  cpg_gp_                   878  cpg_gp.F90
oifs_43r3_model.e  0000000000C2E72E  cpg_                      627  cpg.F90
oifs_43r3_model.e  0000000000C2C638  cpg_drv_                  351  cpg_drv.F90
oifs_43r3_model.e  0000000000BF17C8  gp_model_                 419  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0681
(argv4) batchid: 981
(argv5) wuid: 12198325
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f955c5ea5fbeec38f96b0295a6c402a9 to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0681_2012050100_123_981_12198325.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0681_2012050100_123_981_12198325.zip
ic_ancil_file: ic_ancil_12198325
ifsdata_file: ifsdata_12198325
climate_data_file: clim_data_12198325
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_fe74f80180ad967f040aba5e9dfbc2b0 to: /var/lib/boinc-client/slots/7/ic_ancil_12198325.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12198325.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34d48fe255126ff4ba5db643f0e5f61a to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12198325.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12198325.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12198325.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12198325.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12198325.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12198325.xml
-- Model is restarting --
Adjusting last_iter, 469, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12198325.xml
last_cpu_time: 13908
upload_file_number: 19
last_iter: 455
last_upload: 1641600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479
The child process has been launched with process id: 52428
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:52428:52428] [20230119:195022:1674125422:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:28:22 STEP  456 H= 456:00 +CPU= 39.388
  20:28:46 STEP  457 H= 457:00 +CPU= 23.212
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:31:31 STEP  458 H= 458:00 +CPU= 23.287
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:47:58 STEP  459 H= 459:00 +CPU= 35.224
  20:48:20 STEP  460 H= 460:00 +CPU= 21.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:50:32 STEP  461 H= 461:00 +CPU= 20.919
  20:51:09 STEP  462 H= 462:00 +CPU= 35.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:12:23 STEP  463 H= 463:00 +CPU= 21.663
  21:12:44 STEP  464 H= 464:00 +CPU= 20.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:14:47 STEP  465 H= 465:00 +CPU= 31.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:16:30 STEP  466 H= 466:00 +CPU= 21.705
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:18:02 STEP  467 H= 467:00 +CPU= 20.473
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:19:38 STEP  468 H= 468:00 +CPU= 34.641
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:21:01 STEP  469 H= 469:00 +CPU= 21.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:22:44 STEP  470 H= 470:00 +CPU= 20.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:24:17 STEP  471 H= 471:00 +CPU= 32.438
  21:24:38 STEP  472 H= 472:00 +CPU= 20.658
  21:24:58 STEP  473 H= 473:00 +CPU= 20.278
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:26:50 STEP  474 H= 474:00 +CPU= 31.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:28:43 STEP  475 H= 475:00 +CPU= 20.350
  21:29:04 STEP  476 H= 476:00 +CPU= 20.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:30:57 STEP  477 H= 477:00 +CPU= 31.897
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:32:29 STEP  478 H= 478:00 +CPU= 21.045
  21:32:50 STEP  479 H= 479:00 +CPU= 21.470
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:35:50 STEP  480 H= 480:00 +CPU= 38.271
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_19.zip
Uploading the intermediate file: upload_file_19.zip
  21:36:11 STEP  481 H= 481:00 +CPU= 20.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:37:43 STEP  482 H= 482:00 +CPU= 21.076
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:39:46 STEP  483 H= 483:00 +CPU= 31.348
  21:40:07 STEP  484 H= 484:00 +CPU= 21.292
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:41:39 STEP  485 H= 485:00 +CPU= 20.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:43:32 STEP  486 H= 486:00 +CPU= 33.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:44:53 STEP  487 H= 487:00 +CPU= 20.881
  21:45:14 STEP  488 H= 488:00 +CPU= 20.517
  21:45:44 STEP  489 H= 489:00 +CPU= 30.724
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:47:37 STEP  490 H= 490:00 +CPU= 21.647
  21:47:57 STEP  491 H= 491:00 +CPU= 20.083
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:49:43 STEP  492 H= 492:00 +CPU= 34.644
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:51:14 STEP  493 H= 493:00 +CPU= 20.013
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:52:37 STEP  494 H= 494:00 +CPU= 21.324
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0681
(argv4) batchid: 981
(argv5) wuid: 12198325
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f955c5ea5fbeec38f96b0295a6c402a9 to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0681_2012050100_123_981_12198325.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0681_2012050100_123_981_12198325.zip
ic_ancil_file: ic_ancil_12198325
ifsdata_file: ifsdata_12198325
climate_data_file: clim_data_12198325
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_fe74f80180ad967f040aba5e9dfbc2b0 to: /var/lib/boinc-client/slots/7/ic_ancil_12198325.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12198325.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34d48fe255126ff4ba5db643f0e5f61a to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12198325.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12198325.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12198325.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12198325.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12198325.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12198325.xml
-- Model is restarting --
Adjusting last_iter, 495, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12198325.xml
last_cpu_time: 14901
upload_file_number: 20
last_iter: 479
last_upload: 1728000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479
The child process has been launched with process id: 74095
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:74095:74095] [20230119:215718:1674133038:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:58:59 STEP  480 H= 480:00 +CPU= 36.460
  21:59:20 STEP  481 H= 481:00 +CPU= 20.681
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:01:00 STEP  482 H= 482:00 +CPU= 20.418
  22:01:33 STEP  483 H= 483:00 +CPU= 32.640
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:03:16 STEP  484 H= 484:00 +CPU= 22.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:04:37 STEP  485 H= 485:00 +CPU= 19.875
  22:05:09 STEP  486 H= 486:00 +CPU= 32.400
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:06:41 STEP  487 H= 487:00 +CPU= 20.782
  22:07:02 STEP  488 H= 488:00 +CPU= 20.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:09:07 STEP  489 H= 489:00 +CPU= 32.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:10:59 STEP  490 H= 490:00 +CPU= 21.062
  22:11:20 STEP  491 H= 491:00 +CPU= 20.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:13:47 STEP  492 H= 492:00 +CPU= 36.115
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000492
  22:14:08 STEP  493 H= 493:00 +CPU= 20.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:15:39 STEP  494 H= 494:00 +CPU= 21.304
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:17:13 STEP  495 H= 495:00 +CPU= 32.478
  22:17:33 STEP  496 H= 496:00 +CPU= 20.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:19:05 STEP  497 H= 497:00 +CPU= 20.284
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:21:10 STEP  498 H= 498:00 +CPU= 33.435
  22:21:30 STEP  499 H= 499:00 +CPU= 20.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:23:01 STEP  500 H= 500:00 +CPU= 20.692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:24:44 STEP  501 H= 501:00 +CPU= 32.161
  22:25:05 STEP  502 H= 502:00 +CPU= 20.832
  22:25:26 STEP  503 H= 503:00 +CPU= 20.811
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:47 STEP  504 H= 504:00 +CPU= 38.450
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_20.zip
Uploading the intermediate file: upload_file_20.zip
  22:29:07 STEP  505 H= 505:00 +CPU= 20.227
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:30:49 STEP  506 H= 506:00 +CPU= 20.615
  22:31:21 STEP  507 H= 507:00 +CPU= 32.260
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:32:54 STEP  508 H= 508:00 +CPU= 20.785
  22:33:16 STEP  509 H= 509:00 +CPU= 21.381
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:35:20 STEP  510 H= 510:00 +CPU= 33.364
  22:35:41 STEP  511 H= 511:00 +CPU= 20.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:37:33 STEP  512 H= 512:00 +CPU= 21.439
  22:38:05 STEP  513 H= 513:00 +CPU= 31.633
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:39:57 STEP  514 H= 514:00 +CPU= 21.335
  22:40:18 STEP  515 H= 515:00 +CPU= 20.684
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:42:05 STEP  516 H= 516:00 +CPU= 35.244
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:43:46 STEP  517 H= 517:00 +CPU= 20.474
  22:44:07 STEP  518 H= 518:00 +CPU= 21.077
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:46:21 STEP  519 H= 519:00 +CPU= 32.259
  22:46:42 STEP  520 H= 520:00 +CPU= 20.871
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:48:14 STEP  521 H= 521:00 +CPU= 20.153
  22:48:46 STEP  522 H= 522:00 +CPU= 32.303
  22:49:06 STEP  523 H= 523:00 +CPU= 19.687
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:51:08 STEP  524 H= 524:00 +CPU= 21.002
  22:51:40 STEP  525 H= 525:00 +CPU= 31.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:22 STEP  526 H= 526:00 +CPU= 21.576
  22:53:43 STEP  527 H= 527:00 +CPU= 20.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:55:31 STEP  528 H= 528:00 +CPU= 37.468
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_21.zip
Uploading the intermediate file: upload_file_21.zip
  22:55:50 STEP  529 H= 529:00 +CPU= 19.795
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:40 STEP  530 H= 530:00 +CPU= 20.941
  22:58:11 STEP  531 H= 531:00 +CPU= 31.061
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:02 STEP  532 H= 532:00 +CPU= 20.632
  23:00:23 STEP  533 H= 533:00 +CPU= 20.690
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:02:08 STEP  534 H= 534:00 +CPU= 33.469
  23:02:28 STEP  535 H= 535:00 +CPU= 19.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:04:40 STEP  536 H= 536:00 +CPU= 20.865
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:06:23 STEP  537 H= 537:00 +CPU= 32.260
  23:06:44 STEP  538 H= 538:00 +CPU= 21.143
  23:07:04 STEP  539 H= 539:00 +CPU= 20.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:09:00 STEP  540 H= 540:00 +CPU= 35.330
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000540
  23:09:20 STEP  541 H= 541:00 +CPU= 20.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:11:03 STEP  542 H= 542:00 +CPU= 21.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:12:56 STEP  543 H= 543:00 +CPU= 32.791
  23:13:16 STEP  544 H= 544:00 +CPU= 20.616
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:14:46 STEP  545 H= 545:00 +CPU= 19.926
  23:15:19 STEP  546 H= 546:00 +CPU= 32.711
  23:15:39 STEP  547 H= 547:00 +CPU= 19.868
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:17:31 STEP  548 H= 548:00 +CPU= 21.164
  23:18:03 STEP  549 H= 549:00 +CPU= 31.202
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:19:35 STEP  550 H= 550:00 +CPU= 21.233
  23:19:56 STEP  551 H= 551:00 +CPU= 21.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:21:55 STEP  552 H= 552:00 +CPU= 37.892
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_22.zip
Uploading the intermediate file: upload_file_22.zip
  23:22:15 STEP  553 H= 553:00 +CPU= 19.725
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:23:48 STEP  554 H= 554:00 +CPU= 19.166
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:22 STEP  555 H= 555:00 +CPU= 31.915
  23:25:43 STEP  556 H= 556:00 +CPU= 20.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:27:24 STEP  557 H= 557:00 +CPU= 19.875
  23:27:57 STEP  558 H= 558:00 +CPU= 32.607
  23:28:17 STEP  559 H= 559:00 +CPU= 19.919
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:30:08 STEP  560 H= 560:00 +CPU= 20.737
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:31:43 STEP  561 H= 561:00 +CPU= 32.329
  23:32:04 STEP  562 H= 562:00 +CPU= 20.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:33:25 STEP  563 H= 563:00 +CPU= 20.550
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:35:01 STEP  564 H= 564:00 +CPU= 35.338
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000564
  23:35:20 STEP  565 H= 565:00 +CPU= 19.412
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:37:02 STEP  566 H= 566:00 +CPU= 20.473
  23:37:33 STEP  567 H= 567:00 +CPU= 31.415
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:39:05 STEP  568 H= 568:00 +CPU= 20.494
  23:39:25 STEP  569 H= 569:00 +CPU= 20.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:41:19 STEP  570 H= 570:00 +CPU= 32.261
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:42:40 STEP  571 H= 571:00 +CPU= 20.821
  23:43:01 STEP  572 H= 572:00 +CPU= 20.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:44:54 STEP  573 H= 573:00 +CPU= 31.677
  23:45:15 STEP  574 H= 574:00 +CPU= 20.552
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:46:25 STEP  575 H= 575:00 +CPU= 20.692
  23:47:03 STEP  576 H= 576:00 +CPU= 37.622
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_23.zip
Uploading the intermediate file: upload_file_23.zip
  23:47:22 STEP  577 H= 577:00 +CPU= 18.968
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:49:20 STEP  578 H= 578:00 +CPU= 19.557
  23:49:52 STEP  579 H= 579:00 +CPU= 32.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:51:14 STEP  580 H= 580:00 +CPU= 20.814
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:16 STEP  581 H= 581:00 +CPU= 20.090
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:53:58 STEP  582 H= 582:00 +CPU= 31.885
  23:54:20 STEP  583 H= 583:00 +CPU= 21.161
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:55:50 STEP  584 H= 584:00 +CPU= 20.128
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:57:23 STEP  585 H= 585:00 +CPU= 31.733
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2012050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0681
(argv4) batchid: 981
(argv5) wuid: 12198325
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/7
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f955c5ea5fbeec38f96b0295a6c402a9 to: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0681_2012050100_123_981_12198325.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/7/oifs_43r3_ps_0681_2012050100_123_981_12198325.zip
ic_ancil_file: ic_ancil_12198325
ifsdata_file: ifsdata_12198325
climate_data_file: clim_data_12198325
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_fe74f80180ad967f040aba5e9dfbc2b0 to: /var/lib/boinc-client/slots/7/ic_ancil_12198325.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/7/ic_ancil_12198325.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_34d48fe255126ff4ba5db643f0e5f61a to: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12198325.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/7/ifsdata/ifsdata_12198325.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/7/159l_2/clim_data_12198325.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/7/159l_2/clim_data_12198325.zip
Checking for progress XML file: /var/lib/boinc-client/slots/7/progress_file_12198325.xml
Opened progress file ok : /var/lib/boinc-client/slots/7/progress_file_12198325.xml
-- Model is restarting --
Adjusting last_iter, 585, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/7/progress_file_12198325.xml
last_cpu_time: 17540
upload_file_number: 24
last_iter: 575
last_upload: 2073600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479
The child process has been launched with process id: 91900
Executing the command: /var/lib/boinc-client/slots/7/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91900:91900] [20230119:235931:1674140371:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:01:10 STEP  576 H= 576:00 +CPU= 35.895
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:02:33 STEP  577 H= 577:00 +CPU= 21.307
  00:02:53 STEP  578 H= 578:00 +CPU= 20.565
  00:03:25 STEP  579 H= 579:00 +CPU= 31.248
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:05:17 STEP  580 H= 580:00 +CPU= 20.804
  00:05:36 STEP  581 H= 581:00 +CPU= 19.558
  00:06:08 STEP  582 H= 582:00 +CPU= 31.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:08:00 STEP  583 H= 583:00 +CPU= 20.367
  00:08:21 STEP  584 H= 584:00 +CPU= 20.602
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:10:05 STEP  585 H= 585:00 +CPU= 32.757
  00:10:26 STEP  586 H= 586:00 +CPU= 20.688
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:11:56 STEP  587 H= 587:00 +CPU= 20.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:13:51 STEP  588 H= 588:00 +CPU= 35.183
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000588
  00:14:12 STEP  589 H= 589:00 +CPU= 20.886
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:15:55 STEP  590 H= 590:00 +CPU= 20.976
  00:16:27 STEP  591 H= 591:00 +CPU= 31.923
  00:16:48 STEP  592 H= 592:00 +CPU= 20.529
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:18:19 STEP  593 H= 593:00 +CPU= 20.583
  00:18:51 STEP  594 H= 594:00 +CPU= 32.289
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:20:34 STEP  595 H= 595:00 +CPU= 21.419
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:22:07 STEP  596 H= 596:00 +CPU= 21.314
  00:22:39 STEP  597 H= 597:00 +CPU= 32.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:24:22 STEP  598 H= 598:00 +CPU= 21.754
  00:24:43 STEP  599 H= 599:00 +CPU= 20.628
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:26:52 STEP  600 H= 600:00 +CPU= 37.854
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/7/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198325/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0681_2012050100_123_981_12198325_1_r364429479_24.zip
Uploading the intermediate file: upload_file_24.zip
  00:27:13 STEP  601 H= 601:00 +CPU= 20.590
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net