climateprediction.net home page
Task 22259556

Task 22259556

Name oifs_43r3_ps_0235_1989050100_123_958_12174879_0
Workunit 12174879
Created 21 Dec 2022, 16:01:42 UTC
Sent 11 Jan 2023, 21:13:25 UTC
Report deadline 10 Feb 2023, 21:13:25 UTC
Received 14 Jan 2023, 1:14:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 5 hours 42 min 22 sec
CPU time 4 hours 38 min 50 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,413.25 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,675.97 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
NC client, suspending the child process
Resuming the child process
  03:25:28 STEP  360 H= 360:00 +CPU= 37.845
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_14.zip
Uploading the intermediate file: upload_file_14.zip
  03:25:49 STEP  361 H= 361:00 +CPU= 21.086
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:28:21 STEP  362 H= 362:00 +CPU= 21.079
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:31:15 STEP  363 H= 363:00 +CPU= 32.858
  03:31:36 STEP  364 H= 364:00 +CPU= 20.472
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:34:18 STEP  365 H= 365:00 +CPU= 20.633
  03:34:52 STEP  366 H= 366:00 +CPU= 33.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:38:05 STEP  367 H= 367:00 +CPU= 20.791
  03:38:26 STEP  368 H= 368:00 +CPU= 21.198
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:41:21 STEP  369 H= 369:00 +CPU= 32.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:44:23 STEP  370 H= 370:00 +CPU= 21.184
  03:44:43 STEP  371 H= 371:00 +CPU= 19.768
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:49:21 STEP  372 H= 372:00 +CPU= 36.340
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000372
  03:49:41 STEP  373 H= 373:00 +CPU= 20.368
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:52:23 STEP  374 H= 374:00 +CPU= 20.685
  03:52:56 STEP  375 H= 375:00 +CPU= 32.137
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:56:58 STEP  376 H= 376:00 +CPU= 21.240
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:59:10 STEP  377 H= 377:00 +CPU= 20.962
  03:59:45 STEP  378 H= 378:00 +CPU= 33.943
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:02:36 STEP  379 H= 379:00 +CPU= 20.731
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:05:18 STEP  380 H= 380:00 +CPU= 21.351
  04:05:51 STEP  381 H= 381:00 +CPU= 32.759
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:08:44 STEP  382 H= 382:00 +CPU= 21.310
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:11:47 STEP  383 H= 383:00 +CPU= 21.416
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:16:58 STEP  384 H= 384:00 +CPU= 39.064
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_15.zip
Uploading the intermediate file: upload_file_15.zip
  04:17:18 STEP  385 H= 385:00 +CPU= 20.637
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:20:11 STEP  386 H= 386:00 +CPU= 21.147
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:23:05 STEP  387 H= 387:00 +CPU= 32.760
  04:23:27 STEP  388 H= 388:00 +CPU= 21.635
  04:23:47 STEP  389 H= 389:00 +CPU= 20.377
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:27:12 STEP  390 H= 390:00 +CPU= 33.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:30:13 STEP  391 H= 391:00 +CPU= 19.994
  04:30:36 STEP  392 H= 392:00 +CPU= 22.111
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:34:09 STEP  393 H= 393:00 +CPU= 32.241
  04:34:31 STEP  394 H= 394:00 +CPU= 21.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:37:22 STEP  395 H= 395:00 +CPU= 20.623
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:40:00 STEP  396 H= 396:00 +CPU= 36.104
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:42:50 STEP  397 H= 397:00 +CPU= 20.088
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 958
(argv5) wuid: 12174879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/15
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2e41b1d3e8d413b3053d677cc9576f0b to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0235_1989050100_123_958_12174879.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0235_1989050100_123_958_12174879.zip
ic_ancil_file: ic_ancil_12174879
ifsdata_file: ifsdata_12174879
climate_data_file: clim_data_12174879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7a2dbfcede8190055d844e7794b11a66 to: /var/lib/boinc-client/slots/15/ic_ancil_12174879.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/15/ic_ancil_12174879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4a3dfbfaaa3488d2043b8bfbedee7e65 to: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12174879.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12174879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/15/159l_2/clim_data_12174879.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/15/159l_2/clim_data_12174879.zip
Checking for progress XML file: /var/lib/boinc-client/slots/15/progress_file_12174879.xml
Opened progress file ok : /var/lib/boinc-client/slots/15/progress_file_12174879.xml
-- Model is restarting --
Adjusting last_iter, 398, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/15/progress_file_12174879.xml
last_cpu_time: 12173
upload_file_number: 16
last_iter: 383
last_upload: 1382400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801
The child process has been launched with process id: 1930498
Executing the command: /var/lib/boinc-client/slots/15/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1930498:1930498] [20230114:044516:1673639116:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:48:08 STEP  384 H= 384:00 +CPU= 38.136
  04:48:29 STEP  385 H= 385:00 +CPU= 20.727
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:50:41 STEP  386 H= 386:00 +CPU= 20.928
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:53:34 STEP  387 H= 387:00 +CPU= 32.751
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:56:26 STEP  388 H= 388:00 +CPU= 21.117
  04:56:47 STEP  389 H= 389:00 +CPU= 20.564
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  04:59:43 STEP  390 H= 390:00 +CPU= 34.655
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:03:06 STEP  391 H= 391:00 +CPU= 21.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:05:19 STEP  392 H= 392:00 +CPU= 20.913
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:07:54 STEP  393 H= 393:00 +CPU= 32.846
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:10:06 STEP  394 H= 394:00 +CPU= 21.411
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:11:59 STEP  395 H= 395:00 +CPU= 20.422
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:14:57 STEP  396 H= 396:00 +CPU= 35.437
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000396
  05:15:18 STEP  397 H= 397:00 +CPU= 20.736
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:18:00 STEP  398 H= 398:00 +CPU= 21.661
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:20:46 STEP  399 H= 399:00 +CPU= 33.232
  05:21:07 STEP  400 H= 400:00 +CPU= 20.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:23:29 STEP  401 H= 401:00 +CPU= 20.262
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:26:13 STEP  402 H= 402:00 +CPU= 32.854
  05:26:34 STEP  403 H= 403:00 +CPU= 20.157
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:29:16 STEP  404 H= 404:00 +CPU= 20.940
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:32:11 STEP  405 H= 405:00 +CPU= 33.351
  05:32:33 STEP  406 H= 406:00 +CPU= 21.541
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:35:46 STEP  407 H= 407:00 +CPU= 21.324
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:38:56 STEP  408 H= 408:00 +CPU= 38.628
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_16.zip
Uploading the intermediate file: upload_file_16.zip
  05:39:16 STEP  409 H= 409:00 +CPU= 20.071
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:41:49 STEP  410 H= 410:00 +CPU= 19.487
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:44:54 STEP  411 H= 411:00 +CPU= 33.689
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:47:46 STEP  412 H= 412:00 +CPU= 20.649
  05:48:07 STEP  413 H= 413:00 +CPU= 21.313
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:52:13 STEP  414 H= 414:00 +CPU= 33.822
  05:52:35 STEP  415 H= 415:00 +CPU= 21.088
  05:52:56 STEP  416 H= 416:00 +CPU= 21.292
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:56:50 STEP  417 H= 417:00 +CPU= 32.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:59:33 STEP  418 H= 418:00 +CPU= 20.724
  05:59:54 STEP  419 H= 419:00 +CPU= 21.457
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:54 STEP  420 H= 420:00 +CPU= 36.201
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000420
  06:04:14 STEP  421 H= 421:00 +CPU= 19.643
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:06:47 STEP  422 H= 422:00 +CPU= 20.886
  06:07:19 STEP  423 H= 423:00 +CPU= 32.444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:09:41 STEP  424 H= 424:00 +CPU= 21.016
  06:10:02 STEP  425 H= 425:00 +CPU= 20.280
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:12:46 STEP  426 H= 426:00 +CPU= 33.327
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:14:48 STEP  427 H= 427:00 +CPU= 20.288
  06:15:09 STEP  428 H= 428:00 +CPU= 21.275
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:17:42 STEP  429 H= 429:00 +CPU= 32.559
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:19:35 STEP  430 H= 430:00 +CPU= 21.047
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:21:17 STEP  431 H= 431:00 +CPU= 21.044
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:24:59 STEP  432 H= 432:00 +CPU= 37.997
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_17.zip
Uploading the intermediate file: upload_file_17.zip
  06:25:19 STEP  433 H= 433:00 +CPU= 20.782
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:27:12 STEP  434 H= 434:00 +CPU= 21.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:29:56 STEP  435 H= 435:00 +CPU= 32.276
  06:30:17 STEP  436 H= 436:00 +CPU= 20.697
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:32:59 STEP  437 H= 437:00 +CPU= 20.293
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:35:03 STEP  438 H= 438:00 +CPU= 33.575
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:36:35 STEP  439 H= 439:00 +CPU= 20.994
  06:36:57 STEP  440 H= 440:00 +CPU= 21.148
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:39:10 STEP  441 H= 441:00 +CPU= 32.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:42:12 STEP  442 H= 442:00 +CPU= 20.757
  06:42:33 STEP  443 H= 443:00 +CPU= 21.235
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:45:10 STEP  444 H= 444:00 +CPU= 35.617
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000444
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 958
(argv5) wuid: 12174879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/15
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2e41b1d3e8d413b3053d677cc9576f0b to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0235_1989050100_123_958_12174879.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0235_1989050100_123_958_12174879.zip
ic_ancil_file: ic_ancil_12174879
ifsdata_file: ifsdata_12174879
climate_data_file: clim_data_12174879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7a2dbfcede8190055d844e7794b11a66 to: /var/lib/boinc-client/slots/15/ic_ancil_12174879.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/15/ic_ancil_12174879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4a3dfbfaaa3488d2043b8bfbedee7e65 to: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12174879.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12174879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/15/159l_2/clim_data_12174879.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/15/159l_2/clim_data_12174879.zip
Checking for progress XML file: /var/lib/boinc-client/slots/15/progress_file_12174879.xml
Opened progress file ok : /var/lib/boinc-client/slots/15/progress_file_12174879.xml
-- Model is restarting --
Adjusting last_iter, 445, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/15/progress_file_12174879.xml
last_cpu_time: 13728
upload_file_number: 18
last_iter: 431
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801
The child process has been launched with process id: 1954421
Executing the command: /var/lib/boinc-client/slots/15/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1954421:1954421] [20230114:064800:1673646480:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:50:31 STEP  432 H= 432:00 +CPU= 36.331
  06:50:52 STEP  433 H= 433:00 +CPU= 20.821
  06:51:13 STEP  434 H= 434:00 +CPU= 20.782
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:53:46 STEP  435 H= 435:00 +CPU= 32.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:56:08 STEP  436 H= 436:00 +CPU= 20.863
  06:56:28 STEP  437 H= 437:00 +CPU= 20.521
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:58:52 STEP  438 H= 438:00 +CPU= 33.054
  06:59:13 STEP  439 H= 439:00 +CPU= 20.220
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:01:56 STEP  440 H= 440:00 +CPU= 21.735
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:05:40 STEP  441 H= 441:00 +CPU= 33.281
  07:06:01 STEP  442 H= 442:00 +CPU= 21.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:08:34 STEP  443 H= 443:00 +CPU= 21.003
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:12:22 STEP  444 H= 444:00 +CPU= 36.447
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000444
  07:12:43 STEP  445 H= 445:00 +CPU= 20.507
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:14:56 STEP  446 H= 446:00 +CPU= 20.822
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:17:20 STEP  447 H= 447:00 +CPU= 33.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:20:13 STEP  448 H= 448:00 +CPU= 21.343
  07:20:34 STEP  449 H= 449:00 +CPU= 21.368
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:23:19 STEP  450 H= 450:00 +CPU= 33.627
  07:23:39 STEP  451 H= 451:00 +CPU= 20.024
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:26:13 STEP  452 H= 452:00 +CPU= 21.226
  07:26:45 STEP  453 H= 453:00 +CPU= 32.384
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:29:27 STEP  454 H= 454:00 +CPU= 20.804
  07:29:49 STEP  455 H= 455:00 +CPU= 21.376
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:33:53 STEP  456 H= 456:00 +CPU= 38.440
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_18.zip
Uploading the intermediate file: upload_file_18.zip
  07:34:14 STEP  457 H= 457:00 +CPU= 20.849
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:37:07 STEP  458 H= 458:00 +CPU= 21.351
  07:37:39 STEP  459 H= 459:00 +CPU= 32.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:40:01 STEP  460 H= 460:00 +CPU= 20.734
  07:40:22 STEP  461 H= 461:00 +CPU= 20.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:43:17 STEP  462 H= 462:00 +CPU= 33.656
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:45:29 STEP  463 H= 463:00 +CPU= 20.134
  07:45:50 STEP  464 H= 464:00 +CPU= 21.235
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:48:54 STEP  465 H= 465:00 +CPU= 31.822
  07:49:16 STEP  466 H= 466:00 +CPU= 21.347
  07:49:36 STEP  467 H= 467:00 +CPU= 20.083
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:52:12 STEP  468 H= 468:00 +CPU= 36.160
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:54:15 STEP  469 H= 469:00 +CPU= 20.397
  07:54:36 STEP  470 H= 470:00 +CPU= 21.322
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:57:40 STEP  471 H= 471:00 +CPU= 32.547
  07:58:01 STEP  472 H= 472:00 +CPU= 20.734
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:00:33 STEP  473 H= 473:00 +CPU= 20.381
  08:01:07 STEP  474 H= 474:00 +CPU= 33.270
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:03:48 STEP  475 H= 475:00 +CPU= 19.941
  08:04:10 STEP  476 H= 476:00 +CPU= 21.654
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:06:53 STEP  477 H= 477:00 +CPU= 31.774
  08:07:14 STEP  478 H= 478:00 +CPU= 21.369
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:09:52 STEP  479 H= 479:00 +CPU= 21.477
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:12:23 STEP  480 H= 480:00 +CPU= 38.210
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_19.zip
Uploading the intermediate file: upload_file_19.zip
  08:12:43 STEP  481 H= 481:00 +CPU= 20.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:14:55 STEP  482 H= 482:00 +CPU= 20.740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:17:11 STEP  483 H= 483:00 +CPU= 32.713
  08:17:31 STEP  484 H= 484:00 +CPU= 20.293
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:19:53 STEP  485 H= 485:00 +CPU= 20.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:22:19 STEP  486 H= 486:00 +CPU= 34.333
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:24:01 STEP  487 H= 487:00 +CPU= 20.940
  08:24:23 STEP  488 H= 488:00 +CPU= 22.127
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:26:46 STEP  489 H= 489:00 +CPU= 32.313
  08:27:07 STEP  490 H= 490:00 +CPU= 20.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:29:49 STEP  491 H= 491:00 +CPU= 20.652
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:31:56 STEP  492 H= 492:00 +CPU= 36.166
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:33:58 STEP  493 H= 493:00 +CPU= 20.719
  08:34:19 STEP  494 H= 494:00 +CPU= 20.899
  08:34:52 STEP  495 H= 495:00 +CPU= 32.876
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:37:03 STEP  496 H= 496:00 +CPU= 21.040
  08:37:24 STEP  497 H= 497:00 +CPU= 20.355
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:40:19 STEP  498 H= 498:00 +CPU= 33.048
  08:40:39 STEP  499 H= 499:00 +CPU= 19.683
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:43:21 STEP  500 H= 500:00 +CPU= 20.976
  08:43:54 STEP  501 H= 501:00 +CPU= 33.109
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:45:57 STEP  502 H= 502:00 +CPU= 21.200
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:48:29 STEP  503 H= 503:00 +CPU= 21.095
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0235
(argv4) batchid: 958
(argv5) wuid: 12174879
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/15
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2e41b1d3e8d413b3053d677cc9576f0b to: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0235_1989050100_123_958_12174879.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/15/oifs_43r3_ps_0235_1989050100_123_958_12174879.zip
ic_ancil_file: ic_ancil_12174879
ifsdata_file: ifsdata_12174879
climate_data_file: clim_data_12174879
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_7a2dbfcede8190055d844e7794b11a66 to: /var/lib/boinc-client/slots/15/ic_ancil_12174879.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/15/ic_ancil_12174879.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_4a3dfbfaaa3488d2043b8bfbedee7e65 to: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12174879.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/15/ifsdata/ifsdata_12174879.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/15/159l_2/clim_data_12174879.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/15/159l_2/clim_data_12174879.zip
Checking for progress XML file: /var/lib/boinc-client/slots/15/progress_file_12174879.xml
Opened progress file ok : /var/lib/boinc-client/slots/15/progress_file_12174879.xml
-- Model is restarting --
Adjusting last_iter, 504, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/15/progress_file_12174879.xml
last_cpu_time: 15547
upload_file_number: 20
last_iter: 503
last_upload: 1728000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801
The child process has been launched with process id: 1972486
Executing the command: /var/lib/boinc-client/slots/15/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1972486:1972486] [20230114:085120:1673653880:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  08:51:58 STEP  504 H= 504:00 +CPU= 36.118
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_20.zip
Uploading the intermediate file: upload_file_20.zip
  08:52:16 STEP  505 H= 505:00 +CPU= 18.080
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:54:45 STEP  506 H= 506:00 +CPU= 20.514
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:57:19 STEP  507 H= 507:00 +CPU= 32.952
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:59:12 STEP  508 H= 508:00 +CPU= 21.202
  08:59:32 STEP  509 H= 509:00 +CPU= 20.623
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:02:38 STEP  510 H= 510:00 +CPU= 34.415
  09:02:58 STEP  511 H= 511:00 +CPU= 20.214
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:06:10 STEP  512 H= 512:00 +CPU= 20.540
  09:06:44 STEP  513 H= 513:00 +CPU= 33.439
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:09:06 STEP  514 H= 514:00 +CPU= 21.130
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:11:07 STEP  515 H= 515:00 +CPU= 20.719
  09:11:43 STEP  516 H= 516:00 +CPU= 35.986
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:14:04 STEP  517 H= 517:00 +CPU= 20.430
  09:14:25 STEP  518 H= 518:00 +CPU= 20.936
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:18:11 STEP  519 H= 519:00 +CPU= 32.732
  09:18:33 STEP  520 H= 520:00 +CPU= 21.203
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:20:55 STEP  521 H= 521:00 +CPU= 20.754
  09:21:28 STEP  522 H= 522:00 +CPU= 33.279
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:24:00 STEP  523 H= 523:00 +CPU= 20.882
  09:24:21 STEP  524 H= 524:00 +CPU= 20.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:26:54 STEP  525 H= 525:00 +CPU= 33.084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:29:07 STEP  526 H= 526:00 +CPU= 21.221
  09:29:28 STEP  527 H= 527:00 +CPU= 21.397
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:32:07 STEP  528 H= 528:00 +CPU= 37.688
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174879/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0235_1989050100_123_958_12174879_0_r1638207801_21.zip
Uploading the intermediate file: upload_file_21.zip
  09:32:27 STEP  529 H= 529:00 +CPU= 19.855
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:35:00 STEP  530 H= 530:00 +CPU= 21.124
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:37:34 STEP  531 H= 531:00 +CPU= 32.400
  09:37:56 STEP  532 H= 532:00 +CPU= 21.674
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:39:47 STEP  533 H= 533:00 +CPU= 19.982
  09:40:20 STEP  534 H= 534:00 +CPU= 33.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:42:23 STEP  535 H= 535:00 +CPU= 20.648
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:44:25 STEP  536 H= 536:00 +CPU= 21.307
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:46:49 STEP  537 H= 537:00 +CPU= 33.522
  09:47:11 STEP  538 H= 538:00 +CPU= 21.462
  09:47:31 STEP  539 H= 539:00 +CPU= 19.960
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:50:20 STEP  540 H= 540:00 +CPU= 36.620
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/15/ICMUAhq0f+000540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:52:31 STEP  541 H= 541:00 +CPU= 19.889
  09:52:53 STEP  542 H= 542:00 +CPU= 21.844
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:55:07 STEP  543 H= 543:00 +CPU= 32.231
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:57:20 STEP  544 H= 544:00 +CPU= 21.160
  09:57:40 STEP  545 H= 545:00 +CPU= 20.517
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:00:15 STEP  546 H= 546:00 +CPU= 33.511
  10:00:35 STEP  547 H= 547:00 +CPU= 20.397
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:05:28 STEP  548 H= 548:00 +CPU= 21.225
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:08:12 STEP  549 H= 549:00 +CPU= 32.526
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net