climateprediction.net home page
Task 22257854

Task 22257854

Name oifs_43r3_ps_0655_1987050100_123_956_12173299_0
Workunit 12173299
Created 21 Dec 2022, 11:51:03 UTC
Sent 13 Jan 2023, 18:08:59 UTC
Report deadline 12 Feb 2023, 18:08:59 UTC
Received 15 Jan 2023, 16:57:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 4 hours 0 min 31 sec
CPU time 3 hours 40 min 18 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,414.16 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,688.79 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
IRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2219226:2219226] [20230115:093248:1673742768:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:36:02 STEP  216 H= 216:00 +CPU= 37.530
  09:36:23 STEP  217 H= 217:00 +CPU= 20.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:39:07 STEP  218 H= 218:00 +CPU= 21.999
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:41:51 STEP  219 H= 219:00 +CPU= 33.084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:44:03 STEP  220 H= 220:00 +CPU= 21.015
  09:44:25 STEP  221 H= 221:00 +CPU= 20.987
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:47:30 STEP  222 H= 222:00 +CPU= 34.223
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:49:31 STEP  223 H= 223:00 +CPU= 20.190
  09:49:53 STEP  224 H= 224:00 +CPU= 21.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:52:37 STEP  225 H= 225:00 +CPU= 32.931
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:55:19 STEP  226 H= 226:00 +CPU= 20.534
  09:55:39 STEP  227 H= 227:00 +CPU= 20.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:58:17 STEP  228 H= 228:00 +CPU= 36.764
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000228
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:00:49 STEP  229 H= 229:00 +CPU= 20.386
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:02:42 STEP  230 H= 230:00 +CPU= 21.933
  10:03:15 STEP  231 H= 231:00 +CPU= 33.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:05:38 STEP  232 H= 232:00 +CPU= 21.306
  10:05:59 STEP  233 H= 233:00 +CPU= 20.546
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:08:23 STEP  234 H= 234:00 +CPU= 33.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:10:44 STEP  235 H= 235:00 +CPU= 19.902
  10:11:06 STEP  236 H= 236:00 +CPU= 21.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:14:52 STEP  237 H= 237:00 +CPU= 33.099
  10:15:13 STEP  238 H= 238:00 +CPU= 21.017
  10:15:33 STEP  239 H= 239:00 +CPU= 20.644
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:18:43 STEP  240 H= 240:00 +CPU= 38.535
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000240
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000216
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000228
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000228
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865_9.zip
Uploading the intermediate file: upload_file_9.zip
  10:19:02 STEP  241 H= 241:00 +CPU= 18.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:20:52 STEP  242 H= 242:00 +CPU= 19.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:23:06 STEP  243 H= 243:00 +CPU= 32.579
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:25:29 STEP  244 H= 244:00 +CPU= 21.958
  10:25:50 STEP  245 H= 245:00 +CPU= 20.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:28:35 STEP  246 H= 246:00 +CPU= 33.457
  10:28:55 STEP  247 H= 247:00 +CPU= 20.315
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:31:17 STEP  248 H= 248:00 +CPU= 21.559
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:33:41 STEP  249 H= 249:00 +CPU= 32.960
  10:34:03 STEP  250 H= 250:00 +CPU= 21.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:36:35 STEP  251 H= 251:00 +CPU= 20.992
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:39:04 STEP  252 H= 252:00 +CPU= 36.451
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000252
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:40:37 STEP  253 H= 253:00 +CPU= 20.978
  10:40:58 STEP  254 H= 254:00 +CPU= 20.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:43:33 STEP  255 H= 255:00 +CPU= 33.192
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:45:55 STEP  256 H= 256:00 +CPU= 20.955
  10:46:17 STEP  257 H= 257:00 +CPU= 21.506
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:49:32 STEP  258 H= 258:00 +CPU= 33.611
  10:49:53 STEP  259 H= 259:00 +CPU= 20.876
  10:50:14 STEP  260 H= 260:00 +CPU= 20.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:55:01 STEP  261 H= 261:00 +CPU= 33.339
  10:55:22 STEP  262 H= 262:00 +CPU= 21.220
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:58:35 STEP  263 H= 263:00 +CPU= 20.884
  10:59:14 STEP  264 H= 264:00 +CPU= 38.296
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000264
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000240
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000252
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000252
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865_10.zip
Uploading the intermediate file: upload_file_10.zip
  10:59:33 STEP  265 H= 265:00 +CPU= 18.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:02:26 STEP  266 H= 266:00 +CPU= 20.281
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:05:00 STEP  267 H= 267:00 +CPU= 32.482
  11:05:22 STEP  268 H= 268:00 +CPU= 21.624
  11:05:42 STEP  269 H= 269:00 +CPU= 20.241
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:08:47 STEP  270 H= 270:00 +CPU= 33.565
  11:09:07 STEP  271 H= 271:00 +CPU= 20.386
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:12:11 STEP  272 H= 272:00 +CPU= 21.084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:15:15 STEP  273 H= 273:00 +CPU= 32.858
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:17:37 STEP  274 H= 274:00 +CPU= 20.945
  11:17:58 STEP  275 H= 275:00 +CPU= 20.295
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:21:35 STEP  276 H= 276:00 +CPU= 36.526
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000276
  11:21:56 STEP  277 H= 277:00 +CPU= 20.621
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:25:09 STEP  278 H= 278:00 +CPU= 21.383
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:28:13 STEP  279 H= 279:00 +CPU= 33.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:30:46 STEP  280 H= 280:00 +CPU= 21.341
  11:31:06 STEP  281 H= 281:00 +CPU= 20.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:35:01 STEP  282 H= 282:00 +CPU= 32.639
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0655
(argv4) batchid: 956
(argv5) wuid: 12173299
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e72a01d5b3ec5a374f0ce87ae93cf0c9 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0655_1987050100_123_956_12173299.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0655_1987050100_123_956_12173299.zip
ic_ancil_file: ic_ancil_12173299
ifsdata_file: ifsdata_12173299
climate_data_file: clim_data_12173299
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9d084cc41bdb903e108ac541e4b2896b to: /var/lib/boinc-client/slots/8/ic_ancil_12173299.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12173299.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_747fec1e92b8662c044860858b7d1a18 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12173299.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12173299.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12173299.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12173299.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12173299.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12173299.xml
-- Model is restarting --
Adjusting last_iter, 283, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12173299.xml
last_cpu_time: 10101
upload_file_number: 11
last_iter: 263
last_upload: 950400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865
The child process has been launched with process id: 2237939
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2237939:2237939] [20230115:113953:1673750393:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:37 STEP  264 H= 264:00 +CPU= 37.406
  11:42:57 STEP  265 H= 265:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:30 STEP  266 H= 266:00 +CPU= 21.107
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:34 STEP  267 H= 267:00 +CPU= 33.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:37 STEP  268 H= 268:00 +CPU= 21.032
  11:52:58 STEP  269 H= 269:00 +CPU= 20.734
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:33 STEP  270 H= 270:00 +CPU= 33.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:57:44 STEP  271 H= 271:00 +CPU= 20.416
  11:58:05 STEP  272 H= 272:00 +CPU= 20.626
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:01:09 STEP  273 H= 273:00 +CPU= 32.680
  12:01:30 STEP  274 H= 274:00 +CPU= 21.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:05:33 STEP  275 H= 275:00 +CPU= 21.201
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:09:20 STEP  276 H= 276:00 +CPU= 36.774
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000276
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000276
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:12:52 STEP  277 H= 277:00 +CPU= 20.081
  12:13:14 STEP  278 H= 278:00 +CPU= 21.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:16:41 STEP  279 H= 279:00 +CPU= 34.262
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:19:44 STEP  280 H= 280:00 +CPU= 21.737
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:23:05 STEP  281 H= 281:00 +CPU= 20.183
  12:23:39 STEP  282 H= 282:00 +CPU= 33.731
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:27:11 STEP  283 H= 283:00 +CPU= 19.908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:30:14 STEP  284 H= 284:00 +CPU= 21.370
  12:30:47 STEP  285 H= 285:00 +CPU= 32.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:34:09 STEP  286 H= 286:00 +CPU= 21.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:36:12 STEP  287 H= 287:00 +CPU= 21.854
  12:36:49 STEP  288 H= 288:00 +CPU= 37.456
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000288
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000288
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000264
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000276
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000276
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865_11.zip
Uploading the intermediate file: upload_file_11.zip
  12:37:10 STEP  289 H= 289:00 +CPU= 20.240
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:22:48 STEP  290 H= 290:00 +CPU= 20.947
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:26:42 STEP  291 H= 291:00 +CPU= 32.357
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0655
(argv4) batchid: 956
(argv5) wuid: 12173299
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e72a01d5b3ec5a374f0ce87ae93cf0c9 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0655_1987050100_123_956_12173299.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0655_1987050100_123_956_12173299.zip
ic_ancil_file: ic_ancil_12173299
ifsdata_file: ifsdata_12173299
climate_data_file: clim_data_12173299
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9d084cc41bdb903e108ac541e4b2896b to: /var/lib/boinc-client/slots/8/ic_ancil_12173299.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12173299.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_747fec1e92b8662c044860858b7d1a18 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12173299.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12173299.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12173299.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12173299.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12173299.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12173299.xml
-- Model is restarting --
Adjusting last_iter, 292, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12173299.xml
last_cpu_time: 10826
upload_file_number: 12
last_iter: 287
last_upload: 1036800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865
The child process has been launched with process id: 2360253
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2360253:2360253] [20230115:222907:1673789347:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:32:08 STEP  288 H= 288:00 +CPU= 37.315
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:34:51 STEP  289 H= 289:00 +CPU= 20.070
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:37:14 STEP  290 H= 290:00 +CPU= 20.881
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:39:47 STEP  291 H= 291:00 +CPU= 32.530
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:41:40 STEP  292 H= 292:00 +CPU= 21.036
  22:41:59 STEP  293 H= 293:00 +CPU= 19.642
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:44:53 STEP  294 H= 294:00 +CPU= 32.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:46:55 STEP  295 H= 295:00 +CPU= 20.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:48:57 STEP  296 H= 296:00 +CPU= 20.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:51:10 STEP  297 H= 297:00 +CPU= 32.317
  22:51:30 STEP  298 H= 298:00 +CPU= 20.081
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:52 STEP  299 H= 299:00 +CPU= 19.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:56:19 STEP  300 H= 300:00 +CPU= 35.590
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000300
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000300
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000300
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:58:41 STEP  301 H= 301:00 +CPU= 20.898
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:44 STEP  302 H= 302:00 +CPU= 21.826
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:02:57 STEP  303 H= 303:00 +CPU= 32.364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:05:09 STEP  304 H= 304:00 +CPU= 20.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:07:01 STEP  305 H= 305:00 +CPU= 20.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:09:27 STEP  306 H= 306:00 +CPU= 34.373
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:11:19 STEP  307 H= 307:00 +CPU= 20.258
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:13:23 STEP  308 H= 308:00 +CPU= 21.342
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:15:27 STEP  309 H= 309:00 +CPU= 32.839
  23:15:47 STEP  310 H= 310:00 +CPU= 20.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:18:20 STEP  311 H= 311:00 +CPU= 21.156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:20:38 STEP  312 H= 312:00 +CPU= 38.403
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000312
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000312
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000288
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000300
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000300
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865_12.zip
Uploading the intermediate file: upload_file_12.zip
  23:20:59 STEP  313 H= 313:00 +CPU= 20.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:23:23 STEP  314 H= 314:00 +CPU= 19.431
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:56 STEP  315 H= 315:00 +CPU= 33.045
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:28:27 STEP  316 H= 316:00 +CPU= 20.432
  23:28:48 STEP  317 H= 317:00 +CPU= 20.300
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:31:32 STEP  318 H= 318:00 +CPU= 32.652
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:33:33 STEP  319 H= 319:00 +CPU= 20.449
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:35:55 STEP  320 H= 320:00 +CPU= 20.874
  23:36:28 STEP  321 H= 321:00 +CPU= 32.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:38:40 STEP  322 H= 322:00 +CPU= 20.894
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:40:52 STEP  323 H= 323:00 +CPU= 20.746
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:43:18 STEP  324 H= 324:00 +CPU= 35.946
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000324
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000324
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000324
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:45:08 STEP  325 H= 325:00 +CPU= 19.895
  23:45:28 STEP  326 H= 326:00 +CPU= 20.598
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:48:22 STEP  327 H= 327:00 +CPU= 32.621
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:50:04 STEP  328 H= 328:00 +CPU= 21.084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:06 STEP  329 H= 329:00 +CPU= 20.210
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:11 STEP  330 H= 330:00 +CPU= 32.989
  23:54:31 STEP  331 H= 331:00 +CPU= 20.028
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:56:43 STEP  332 H= 332:00 +CPU= 21.430
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:59:06 STEP  333 H= 333:00 +CPU= 31.383
  23:59:26 STEP  334 H= 334:00 +CPU= 20.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:01:48 STEP  335 H= 335:00 +CPU= 20.230
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:04:57 STEP  336 H= 336:00 +CPU= 37.793
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000336
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000336
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000312
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000324
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000324
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865_13.zip
Uploading the intermediate file: upload_file_13.zip
  00:05:16 STEP  337 H= 337:00 +CPU= 19.245
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:07:48 STEP  338 H= 338:00 +CPU= 20.566
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:10:40 STEP  339 H= 339:00 +CPU= 31.151
  00:11:01 STEP  340 H= 340:00 +CPU= 20.659
  00:11:21 STEP  341 H= 341:00 +CPU= 19.887
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:14:05 STEP  342 H= 342:00 +CPU= 32.479
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:16:27 STEP  343 H= 343:00 +CPU= 19.954
  00:16:47 STEP  344 H= 344:00 +CPU= 20.552
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:19:30 STEP  345 H= 345:00 +CPU= 32.156
  00:19:51 STEP  346 H= 346:00 +CPU= 20.428
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:21:53 STEP  347 H= 347:00 +CPU= 20.131
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:24:09 STEP  348 H= 348:00 +CPU= 35.023
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000348
  00:24:29 STEP  349 H= 349:00 +CPU= 19.608
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:27:01 STEP  350 H= 350:00 +CPU= 20.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:29:04 STEP  351 H= 351:00 +CPU= 31.918
  00:29:24 STEP  352 H= 352:00 +CPU= 20.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1987050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0655
(argv4) batchid: 956
(argv5) wuid: 12173299
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e72a01d5b3ec5a374f0ce87ae93cf0c9 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0655_1987050100_123_956_12173299.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0655_1987050100_123_956_12173299.zip
ic_ancil_file: ic_ancil_12173299
ifsdata_file: ifsdata_12173299
climate_data_file: clim_data_12173299
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_9d084cc41bdb903e108ac541e4b2896b to: /var/lib/boinc-client/slots/8/ic_ancil_12173299.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12173299.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_747fec1e92b8662c044860858b7d1a18 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12173299.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12173299.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12173299.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12173299.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12173299.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12173299.xml
-- Model is restarting --
Adjusting last_iter, 353, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12173299.xml
last_cpu_time: 12453
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865
The child process has been launched with process id: 2377664
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2377664:2377664] [20230116:003350:1673796830:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:35:58 STEP  336 H= 336:00 +CPU= 35.286
  00:36:18 STEP  337 H= 337:00 +CPU= 19.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:39:08 STEP  338 H= 338:00 +CPU= 20.034
  00:39:39 STEP  339 H= 339:00 +CPU= 31.231
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:42:01 STEP  340 H= 340:00 +CPU= 20.232
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:43:52 STEP  341 H= 341:00 +CPU= 19.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:46:15 STEP  342 H= 342:00 +CPU= 32.495
  00:46:35 STEP  343 H= 343:00 +CPU= 19.635
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:48:38 STEP  344 H= 344:00 +CPU= 20.620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:51:01 STEP  345 H= 345:00 +CPU= 31.883
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:52:34 STEP  346 H= 346:00 +CPU= 20.861
  00:52:53 STEP  347 H= 347:00 +CPU= 19.636
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:55:30 STEP  348 H= 348:00 +CPU= 34.985
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000348
  00:55:50 STEP  349 H= 349:00 +CPU= 20.009
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:58:21 STEP  350 H= 350:00 +CPU= 20.094
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:00:35 STEP  351 H= 351:00 +CPU= 31.762
  01:00:55 STEP  352 H= 352:00 +CPU= 20.513
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:03:37 STEP  353 H= 353:00 +CPU= 19.100
  01:04:09 STEP  354 H= 354:00 +CPU= 32.528
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:06:30 STEP  355 H= 355:00 +CPU= 19.786
  01:06:50 STEP  356 H= 356:00 +CPU= 20.214
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:10:02 STEP  357 H= 357:00 +CPU= 31.305
  01:10:22 STEP  358 H= 358:00 +CPU= 20.267
  01:10:42 STEP  359 H= 359:00 +CPU= 19.627
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:13:30 STEP  360 H= 360:00 +CPU= 36.997
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12173299/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0655_1987050100_123_956_12173299_0_r1826597865_14.zip
Uploading the intermediate file: upload_file_14.zip
  01:15:01 STEP  361 H= 361:00 +CPU= 20.198
  01:15:21 STEP  362 H= 362:00 +CPU= 19.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:18:35 STEP  363 H= 363:00 +CPU= 32.231
  01:18:55 STEP  364 H= 364:00 +CPU= 20.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:21:27 STEP  365 H= 365:00 +CPU= 19.467
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net