<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 0108
(argv4) batchid: 993
(argv5) wuid: 12214742
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_12214742
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb864bcd7efe40dab73d1f7bb623371d to: /var/lib/boinc-client/slots/19/oifs_43r3_0108_2019110100_123_993_12214742.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_0108_2019110100_123_993_12214742.zip
ic_ancil_file: ic_ancil_12214742
ifsdata_file: ifsdata_12214742
climate_data_file: clim_data_12214742
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_4311dc2038f8e9b007deffd08bd85292 to: /var/lib/boinc-client/slots/19/ic_ancil_12214742.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12214742.zip
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_3751d43b7d4a622671e6277a6c523f42 to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12214742.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12214742.zip
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12214742.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12214742.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12214742.xml
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12214742.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_0108_2019110100_123_993_12214742_0_r1072635843
The child process has been launched with process id: 1860855
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:wolta:1:1:1860855:1860855] [20230224:101633:1677255393:0.001] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:18:53 STEP 0 H= 0:00 +CPU= 90.645
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhnw4+000000
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhnw4+000000
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhnw4+000000
10:19:53 STEP 1 H= 1:00 +CPU= 58.565
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:21:12 STEP 2 H= 2:00 +CPU= 55.782
10:22:33 STEP 3 H= 3:00 +CPU= 80.457
10:23:27 STEP 4 H= 4:00 +CPU= 53.776
10:24:15 STEP 5 H= 5:00 +CPU= 48.530
10:25:35 STEP 6 H= 6:00 +CPU= 79.457
10:26:22 STEP 7 H= 7:00 +CPU= 47.093
10:27:12 STEP 8 H= 8:00 +CPU= 49.876
10:28:30 STEP 9 H= 9:00 +CPU= 78.062
10:29:20 STEP 10 H= 10:00 +CPU= 49.691
10:30:08 STEP 11 H= 11:00 +CPU= 47.897
10:31:34 STEP 12 H= 12:00 +CPU= 86.321
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhnw4+000012
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhnw4+000012
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhnw4+000012
10:32:22 STEP 13 H= 13:00 +CPU= 47.772
10:33:03 STEP 14 H= 14:00 +CPU= 41.027
10:34:19 STEP 15 H= 15:00 +CPU= 75.240
10:35:10 STEP 16 H= 16:00 +CPU= 50.868
10:36:09 STEP 17 H= 17:00 +CPU= 51.851
..The child process has been killed with signal: 9
>>> Printing last 70 lines from file: NODE.001_01
SUECSO4: USE AVAILABLE FILES FOR DECADES: 2010 2020
SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE: 2.322061922655761E-003
NSTEP = 15 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.174925206663E-01 1.937140557074E-01 6.781086765457E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.663804089806E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -4.03279680579456E-03 -1.96949467225788E+00 2.21001009246471E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -5.81544802039122E-03 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101141500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101143000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101144500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101150000 !!!!!!!!!!!!
10:34:19 STEP 15 H= 15:00 +CPU= 75.240
NSTEP = 16 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.108151032082E-01 1.924148229556E-01 6.783149750866E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.548566796544E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -6.83867016205741E-03 -1.93108361824613E+00 2.28979241303233E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -7.45209164870038E-03 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101151500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101153000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101154500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101160000 !!!!!!!!!!!!
10:35:10 STEP 16 H= 16:00 +CPU= 50.868
NSTEP = 17 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 5.004271234930E-01 1.918221838753E-01 6.843672864026E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.318433676784E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -1.68112338809770E-02 -2.11888839814002E+00 1.91170012728787E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -1.28412652977725E-02 -1.00000000000000E+00 1.00000000000000E+00
STEPO: evolving YSPSDT_AR1( 1)
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101161500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101163000 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101164500 !!!!!!!!!!!!
!!!!!!!!!!!!!! WAVE FIELDS INTEGRATED. DATE IS: 20191101170000 !!!!!!!!!!!!
10:36:09 STEP 17 H= 17:00 +CPU= 51.851
DATE= 2019 11 1
ISTEP= 18: Reset RSOLINC= 1361.0304 RIP0= 1382.1494
SU_GHGCLIM, RCARDI,RCH4,RN2O,RCFC11,RCFC12,RNO2: 6.240006412635001E-004
9.628461857022168E-007 4.996611132045250E-007 9.607432727722499E-010
2.027017283006666E-009 7.940283000000001E-011
SUECAEC: GLOBAL AVERAGE VOLCANIC SULPHATE FOR (KINDAT,KMINUT) 20191101
1170 9.999999999999998E-005
SUECSO4: USE AVAILABLE FILES FOR DECADES: 2010 2020
SUBROUTINE SUECSO4:, UNWEIGHTED AVERAGE: 2.321359214467334E-003
NSTEP = 18 STEPO 0AAA00AAA
SPECTRAL NORMS: Pattern 1
STEPO, YSPSDT_AR1 ( 1,...), SP NORMS 2D: 4.815536204860E-01 1.922578847709E-01 6.886414357882E-02
STEPO, YSPSDT_AR1 (SUM,...), SP NORMS 2D: 5.233424275221E-01
STEPO, YGPSDT ( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -7.34612685395599E-03 -1.90872358855232E+00 2.04097082428994E+00
STEPO, YGPSDT[clipped]( 1,...), Grid Point Norms
FIELD LEV AVERAGE MINIMUM MAXIMUM
101 60 -6.58127563340051E-03 -1.00000000000000E+00 1.00000000000000E+00
------------------------------------------------
CNT0 not found; string returned was: 'STEPO'
>>> Printing last 8 lines from file: ifs.stat
10:30:14 0AAA00AAA STEPO 12 3.079 3.079 3.085 12:30 13:40 0.21599197788816E-04 2GB 0MB
10:31:34 0AAA00AAA STEPO 13 80.600 80.600 80.768 13:50 15:01 0.21599197788816E-04 2GB 0MB
10:32:22 0AAA00AAA STEPO 14 47.772 47.772 47.853 14:38 15:48 0.21599197788816E-04 2GB 0MB
10:33:03 0AAA00AAA STEPO 15 41.028 41.028 41.103 15:19 16:30 0.21599197788816E-04 2GB 0MB
10:34:19 0AAA00AAA STEPO 16 75.238 75.238 75.372 16:34 17:45 0.21599197788816E-04 2GB 0MB
10:35:10 0AAA00AAA STEPO 17 50.869 50.869 51.439 17:25 18:36 0.21599197788816E-04 2GB 0MB
10:36:09 0AAA00AAA STEPO 18 52.079 52.079 58.834 18:17 19:35 0.21599197788816E-04 2GB 0MB
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/19/progress_file_12214742.xml
<running_values>
<last_cpu_time>1099.530000</last_cpu_time>
<upload_file_number>0</upload_file_number>
<last_iter>18</last_iter>
<last_upload>0</last_upload>
<model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully
</stderr_txt>
]]>
