<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ile: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
last_cpu_time: 10179
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947
The child process has been launched with process id: 2197780
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2197780:2197780] [20230115:074427:1673736267:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:46:58 STEP 336 H= 336:00 +CPU= 37.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:50:11 STEP 337 H= 337:00 +CPU= 21.228
07:50:33 STEP 338 H= 338:00 +CPU= 21.577
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:55:09 STEP 339 H= 339:00 +CPU= 33.587
07:55:30 STEP 340 H= 340:00 +CPU= 21.096
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
07:59:52 STEP 341 H= 341:00 +CPU= 20.492
08:00:28 STEP 342 H= 342:00 +CPU= 34.998
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:04:30 STEP 343 H= 343:00 +CPU= 21.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:07:42 STEP 344 H= 344:00 +CPU= 21.103
08:08:18 STEP 345 H= 345:00 +CPU= 35.993
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:13:12 STEP 346 H= 346:00 +CPU= 22.624
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:16:54 STEP 347 H= 347:00 +CPU= 21.051
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:21:21 STEP 348 H= 348:00 +CPU= 36.013
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000348
08:21:43 STEP 349 H= 349:00 +CPU= 21.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:25:06 STEP 350 H= 350:00 +CPU= 21.460
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:29:22 STEP 351 H= 351:00 +CPU= 33.564
08:29:45 STEP 352 H= 352:00 +CPU= 22.342
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:41:30 STEP 353 H= 353:00 +CPU= 23.172
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:44:56 STEP 354 H= 354:00 +CPU= 33.531
08:45:16 STEP 355 H= 355:00 +CPU= 20.226
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:47:40 STEP 356 H= 356:00 +CPU= 21.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:50:03 STEP 357 H= 357:00 +CPU= 31.933
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:52:06 STEP 358 H= 358:00 +CPU= 21.069
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:54:08 STEP 359 H= 359:00 +CPU= 20.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:56:36 STEP 360 H= 360:00 +CPU= 37.252
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947_14.zip
Uploading the intermediate file: upload_file_14.zip
08:56:57 STEP 361 H= 361:00 +CPU= 20.860
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
08:59:05 STEP 362 H= 362:00 +CPU= 20.319
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:01:39 STEP 363 H= 363:00 +CPU= 31.853
09:02:00 STEP 364 H= 364:00 +CPU= 21.122
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:06:52 STEP 365 H= 365:00 +CPU= 20.001
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:09:16 STEP 366 H= 366:00 +CPU= 33.111
09:09:38 STEP 367 H= 367:00 +CPU= 21.516
09:09:59 STEP 368 H= 368:00 +CPU= 21.229
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:13:25 STEP 369 H= 369:00 +CPU= 33.115
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:16:18 STEP 370 H= 370:00 +CPU= 21.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:29:11 STEP 371 H= 371:00 +CPU= 20.591
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:32:49 STEP 372 H= 372:00 +CPU= 35.897
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000372
09:33:11 STEP 373 H= 373:00 +CPU= 21.451
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:36:03 STEP 374 H= 374:00 +CPU= 21.066
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:38:58 STEP 375 H= 375:00 +CPU= 33.074
09:39:19 STEP 376 H= 376:00 +CPU= 21.429
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:41:51 STEP 377 H= 377:00 +CPU= 20.373
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:44:15 STEP 378 H= 378:00 +CPU= 33.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:47:07 STEP 379 H= 379:00 +CPU= 20.905
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0512
(argv4) batchid: 976
(argv5) wuid: 12193156
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82409068b50dff9c5b858dee342bc04b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0512_2007050100_123_976_12193156.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0512_2007050100_123_976_12193156.zip
ic_ancil_file: ic_ancil_12193156
ifsdata_file: ifsdata_12193156
climate_data_file: clim_data_12193156
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b04eec7a2d95077141dbb8ae3487e804 to: /var/lib/boinc-client/slots/1/ic_ancil_12193156.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12193156.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5e4d1fcf6f824b5f8f1858e4c8e9bfa8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12193156.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12193156.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12193156.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12193156.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12193156.xml
-- Model is restarting --
Adjusting last_iter, 380, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
last_cpu_time: 11312
upload_file_number: 15
last_iter: 359
last_upload: 1296000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947
The child process has been launched with process id: 2221723
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2221723:2221723] [20230115:094931:1673743771:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:52:23 STEP 360 H= 360:00 +CPU= 36.990
09:52:44 STEP 361 H= 361:00 +CPU= 20.399
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:55:26 STEP 362 H= 362:00 +CPU= 21.191
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
09:58:00 STEP 363 H= 363:00 +CPU= 32.787
09:58:22 STEP 364 H= 364:00 +CPU= 21.797
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:00:53 STEP 365 H= 365:00 +CPU= 20.414
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:02:59 STEP 366 H= 366:00 +CPU= 34.039
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:05:21 STEP 367 H= 367:00 +CPU= 20.716
10:05:43 STEP 368 H= 368:00 +CPU= 21.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:08:07 STEP 369 H= 369:00 +CPU= 32.781
10:08:28 STEP 370 H= 370:00 +CPU= 21.360
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:10:50 STEP 371 H= 371:00 +CPU= 20.449
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:12:57 STEP 372 H= 372:00 +CPU= 36.284
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000372
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:14:59 STEP 373 H= 373:00 +CPU= 20.608
10:15:20 STEP 374 H= 374:00 +CPU= 20.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:18:23 STEP 375 H= 375:00 +CPU= 32.068
10:18:45 STEP 376 H= 376:00 +CPU= 21.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:20:56 STEP 377 H= 377:00 +CPU= 20.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:23:11 STEP 378 H= 378:00 +CPU= 33.382
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:25:32 STEP 379 H= 379:00 +CPU= 20.286
10:25:54 STEP 380 H= 380:00 +CPU= 20.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:28:38 STEP 381 H= 381:00 +CPU= 32.921
10:28:59 STEP 382 H= 382:00 +CPU= 21.080
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:31:22 STEP 383 H= 383:00 +CPU= 21.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:33:51 STEP 384 H= 384:00 +CPU= 37.898
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947_15.zip
Uploading the intermediate file: upload_file_15.zip
10:34:11 STEP 385 H= 385:00 +CPU= 20.075
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:36:32 STEP 386 H= 386:00 +CPU= 19.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:38:56 STEP 387 H= 387:00 +CPU= 31.871
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:40:30 STEP 388 H= 388:00 +CPU= 21.845
10:40:51 STEP 389 H= 389:00 +CPU= 20.617
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:43:25 STEP 390 H= 390:00 +CPU= 33.154
10:43:45 STEP 391 H= 391:00 +CPU= 20.410
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:46:08 STEP 392 H= 392:00 +CPU= 21.262
10:46:40 STEP 393 H= 393:00 +CPU= 31.857
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:49:44 STEP 394 H= 394:00 +CPU= 21.524
10:50:04 STEP 395 H= 395:00 +CPU= 20.049
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:53:01 STEP 396 H= 396:00 +CPU= 36.083
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:55:13 STEP 397 H= 397:00 +CPU= 20.294
10:55:34 STEP 398 H= 398:00 +CPU= 21.559
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
10:58:59 STEP 399 H= 399:00 +CPU= 32.925
10:59:20 STEP 400 H= 400:00 +CPU= 20.949
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:02:32 STEP 401 H= 401:00 +CPU= 20.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:05:06 STEP 402 H= 402:00 +CPU= 32.726
11:05:28 STEP 403 H= 403:00 +CPU= 20.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:08:21 STEP 404 H= 404:00 +CPU= 20.817
11:08:54 STEP 405 H= 405:00 +CPU= 32.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:11:56 STEP 406 H= 406:00 +CPU= 20.851
11:12:17 STEP 407 H= 407:00 +CPU= 21.357
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:15:26 STEP 408 H= 408:00 +CPU= 38.285
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947_16.zip
Uploading the intermediate file: upload_file_16.zip
11:15:45 STEP 409 H= 409:00 +CPU= 19.255
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:17:48 STEP 410 H= 410:00 +CPU= 20.193
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:21:23 STEP 411 H= 411:00 +CPU= 32.448
11:21:44 STEP 412 H= 412:00 +CPU= 21.237
11:22:04 STEP 413 H= 413:00 +CPU= 19.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:28:00 STEP 414 H= 414:00 +CPU= 32.667
11:28:22 STEP 415 H= 415:00 +CPU= 21.193
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:30:54 STEP 416 H= 416:00 +CPU= 20.962
11:31:26 STEP 417 H= 417:00 +CPU= 32.065
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:35:09 STEP 418 H= 418:00 +CPU= 21.071
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:37:51 STEP 419 H= 419:00 +CPU= 20.618
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:40:10 STEP 420 H= 420:00 +CPU= 36.898
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000420
11:42:33 STEP 421 H= 421:00 +CPU= 20.877
11:42:54 STEP 422 H= 422:00 +CPU= 20.974
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:46:39 STEP 423 H= 423:00 +CPU= 33.357
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:49:31 STEP 424 H= 424:00 +CPU= 20.709
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0512
(argv4) batchid: 976
(argv5) wuid: 12193156
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82409068b50dff9c5b858dee342bc04b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0512_2007050100_123_976_12193156.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0512_2007050100_123_976_12193156.zip
ic_ancil_file: ic_ancil_12193156
ifsdata_file: ifsdata_12193156
climate_data_file: clim_data_12193156
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b04eec7a2d95077141dbb8ae3487e804 to: /var/lib/boinc-client/slots/1/ic_ancil_12193156.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12193156.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5e4d1fcf6f824b5f8f1858e4c8e9bfa8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12193156.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12193156.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12193156.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12193156.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12193156.xml
-- Model is restarting --
Adjusting last_iter, 425, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
last_cpu_time: 12956
upload_file_number: 17
last_iter: 407
last_upload: 1468800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947
The child process has been launched with process id: 2240038
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2240038:2240038] [20230115:115233:1673751153:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:55:14 STEP 408 H= 408:00 +CPU= 37.355
11:55:35 STEP 409 H= 409:00 +CPU= 20.421
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
11:57:47 STEP 410 H= 410:00 +CPU= 21.021
11:58:19 STEP 411 H= 411:00 +CPU= 31.745
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:01:12 STEP 412 H= 412:00 +CPU= 21.767
12:01:33 STEP 413 H= 413:00 +CPU= 20.710
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:05:49 STEP 414 H= 414:00 +CPU= 34.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:09:20 STEP 415 H= 415:00 +CPU= 20.957
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:12:51 STEP 416 H= 416:00 +CPU= 19.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:16:17 STEP 417 H= 417:00 +CPU= 34.049
12:16:40 STEP 418 H= 418:00 +CPU= 22.251
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:19:41 STEP 419 H= 419:00 +CPU= 20.384
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:23:20 STEP 420 H= 420:00 +CPU= 36.034
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000420
12:23:40 STEP 421 H= 421:00 +CPU= 20.403
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:27:13 STEP 422 H= 422:00 +CPU= 20.587
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:30:27 STEP 423 H= 423:00 +CPU= 33.187
12:30:49 STEP 424 H= 424:00 +CPU= 21.199
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:34:11 STEP 425 H= 425:00 +CPU= 20.707
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:36:24 STEP 426 H= 426:00 +CPU= 32.458
12:36:44 STEP 427 H= 427:00 +CPU= 19.403
12:37:04 STEP 428 H= 428:00 +CPU= 19.943
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:22:55 STEP 429 H= 429:00 +CPU= 32.249
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:26:37 STEP 430 H= 430:00 +CPU= 21.020
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:29:09 STEP 431 H= 431:00 +CPU= 20.463
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:32:09 STEP 432 H= 432:00 +CPU= 38.774
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947_17.zip
Uploading the intermediate file: upload_file_17.zip
22:32:27 STEP 433 H= 433:00 +CPU= 17.736
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:34:49 STEP 434 H= 434:00 +CPU= 19.750
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2007050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0512
(argv4) batchid: 976
(argv5) wuid: 12193156
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_82409068b50dff9c5b858dee342bc04b to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0512_2007050100_123_976_12193156.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0512_2007050100_123_976_12193156.zip
ic_ancil_file: ic_ancil_12193156
ifsdata_file: ifsdata_12193156
climate_data_file: clim_data_12193156
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_b04eec7a2d95077141dbb8ae3487e804 to: /var/lib/boinc-client/slots/1/ic_ancil_12193156.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12193156.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_5e4d1fcf6f824b5f8f1858e4c8e9bfa8 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12193156.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12193156.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12193156.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12193156.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12193156.xml
-- Model is restarting --
Adjusting last_iter, 435, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12193156.xml
last_cpu_time: 13642
upload_file_number: 18
last_iter: 431
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947
The child process has been launched with process id: 2361333
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:2361333:2361333] [20230115:223708:1673789828:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:39:50 STEP 432 H= 432:00 +CPU= 38.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:41:42 STEP 433 H= 433:00 +CPU= 20.752
22:42:02 STEP 434 H= 434:00 +CPU= 20.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:44:56 STEP 435 H= 435:00 +CPU= 32.093
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:46:58 STEP 436 H= 436:00 +CPU= 20.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:48:59 STEP 437 H= 437:00 +CPU= 20.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:51:12 STEP 438 H= 438:00 +CPU= 32.067
22:51:31 STEP 439 H= 439:00 +CPU= 19.574
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:53:54 STEP 440 H= 440:00 +CPU= 20.964
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:56:16 STEP 441 H= 441:00 +CPU= 31.637
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:58:39 STEP 442 H= 442:00 +CPU= 21.583
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:00:41 STEP 443 H= 443:00 +CPU= 20.761
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:02:57 STEP 444 H= 444:00 +CPU= 35.529
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:05:08 STEP 445 H= 445:00 +CPU= 19.726
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:07:00 STEP 446 H= 446:00 +CPU= 21.005
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:09:23 STEP 447 H= 447:00 +CPU= 31.928
23:09:44 STEP 448 H= 448:00 +CPU= 20.971
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:11:37 STEP 449 H= 449:00 +CPU= 20.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:15:24 STEP 450 H= 450:00 +CPU= 32.978
23:15:45 STEP 451 H= 451:00 +CPU= 20.603
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:18:18 STEP 452 H= 452:00 +CPU= 20.618
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:20:32 STEP 453 H= 453:00 +CPU= 32.858
23:20:53 STEP 454 H= 454:00 +CPU= 21.481
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:23:27 STEP 455 H= 455:00 +CPU= 21.010
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:26:05 STEP 456 H= 456:00 +CPU= 37.529
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12193156/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0512_2007050100_123_976_12193156_0_r91305947_18.zip
Uploading the intermediate file: upload_file_18.zip
23:26:23 STEP 457 H= 457:00 +CPU= 17.998
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:28:38 STEP 458 H= 458:00 +CPU= 20.510
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:31:21 STEP 459 H= 459:00 +CPU= 31.844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:33:23 STEP 460 H= 460:00 +CPU= 20.496
23:33:44 STEP 461 H= 461:00 +CPU= 20.985
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:36:18 STEP 462 H= 462:00 +CPU= 32.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:38:29 STEP 463 H= 463:00 +CPU= 20.326
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:40:43 STEP 464 H= 464:00 +CPU= 22.159
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:43:07 STEP 465 H= 465:00 +CPU= 33.302
23:43:28 STEP 466 H= 466:00 +CPU= 21.324
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:45:19 STEP 467 H= 467:00 +CPU= 19.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:48:16 STEP 468 H= 468:00 +CPU= 35.200
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+000468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:49:58 STEP 469 H= 469:00 +CPU= 20.299
23:50:19 STEP 470 H= 470:00 +CPU= 21.189
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:54:06 STEP 471 H= 471:00 +CPU= 32.335
23:54:27 STEP 472 H= 472:00 +CPU= 21.282
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
23:56:39 STEP 473 H= 473:00 +CPU= 20.331
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully
</stderr_txt>
]]>
