climateprediction.net home page
Task 22267346

Task 22267346

Name oifs_43r3_ps_0611_1996050100_123_965_12182255_0
Workunit 12182255
Created 22 Dec 2022, 11:25:45 UTC
Sent 17 Jan 2023, 21:32:05 UTC
Report deadline 18 Mar 2023, 21:32:05 UTC
Received 19 Jan 2023, 17:49:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 5 hours 37 min 33 sec
CPU time 5 hours 23 min 53 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.27 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,682.42 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
limate_data_file: clim_data_12182255
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dffdd5bebc1f978cf06cbfc508d2c0bb to: /var/lib/boinc-client/slots/9/ic_ancil_12182255.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12182255.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a444f6d310b278d4e8f286702082644e to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12182255.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12182255.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12182255.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12182255.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12182255.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12182255.xml
-- Model is restarting --
Adjusting last_iter, 350, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12182255.xml
last_cpu_time: 13204
upload_file_number: 14
last_iter: 335
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017
The child process has been launched with process id: 73450
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:73450:73450] [20230119:215237:1674132757:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:54:27 STEP  336 H= 336:00 +CPU= 36.448
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:55:59 STEP  337 H= 337:00 +CPU= 20.799
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:57:10 STEP  338 H= 338:00 +CPU= 20.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  21:58:43 STEP  339 H= 339:00 +CPU= 31.589
  21:59:04 STEP  340 H= 340:00 +CPU= 21.039
  21:59:24 STEP  341 H= 341:00 +CPU= 19.926
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:01:18 STEP  342 H= 342:00 +CPU= 32.472
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:02:59 STEP  343 H= 343:00 +CPU= 19.924
  22:03:20 STEP  344 H= 344:00 +CPU= 21.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:04:55 STEP  345 H= 345:00 +CPU= 32.797
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:06:28 STEP  346 H= 346:00 +CPU= 20.325
  22:06:49 STEP  347 H= 347:00 +CPU= 21.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:08:56 STEP  348 H= 348:00 +CPU= 35.198
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000348
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000348
  22:09:17 STEP  349 H= 349:00 +CPU= 20.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:11:10 STEP  350 H= 350:00 +CPU= 21.547
  22:11:42 STEP  351 H= 351:00 +CPU= 32.600
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:13:55 STEP  352 H= 352:00 +CPU= 21.659
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:15:27 STEP  353 H= 353:00 +CPU= 20.569
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:16:59 STEP  354 H= 354:00 +CPU= 32.118
  22:17:20 STEP  355 H= 355:00 +CPU= 20.427
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:18:53 STEP  356 H= 356:00 +CPU= 20.906
  22:19:24 STEP  357 H= 357:00 +CPU= 31.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:21:17 STEP  358 H= 358:00 +CPU= 21.637
  22:21:38 STEP  359 H= 359:00 +CPU= 21.071
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:23:27 STEP  360 H= 360:00 +CPU= 37.334
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000360
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000336
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000348
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000348
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_14.zip
Uploading the intermediate file: upload_file_14.zip
  22:24:58 STEP  361 H= 361:00 +CPU= 19.940
  22:25:18 STEP  362 H= 362:00 +CPU= 20.568
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:27:32 STEP  363 H= 363:00 +CPU= 32.431
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:54 STEP  364 H= 364:00 +CPU= 21.086
  22:29:14 STEP  365 H= 365:00 +CPU= 19.934
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:31:08 STEP  366 H= 366:00 +CPU= 32.801
  22:31:29 STEP  367 H= 367:00 +CPU= 20.308
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:33:01 STEP  368 H= 368:00 +CPU= 21.210
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:35:05 STEP  369 H= 369:00 +CPU= 32.430
  22:35:26 STEP  370 H= 370:00 +CPU= 21.034
  22:35:46 STEP  371 H= 371:00 +CPU= 20.093
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:37:53 STEP  372 H= 372:00 +CPU= 36.055
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000372
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000372
  22:38:13 STEP  373 H= 373:00 +CPU= 19.739
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:40:05 STEP  374 H= 374:00 +CPU= 20.620
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:41:49 STEP  375 H= 375:00 +CPU= 32.305
  22:42:09 STEP  376 H= 376:00 +CPU= 20.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:43:50 STEP  377 H= 377:00 +CPU= 20.040
  22:44:23 STEP  378 H= 378:00 +CPU= 32.807
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:46:25 STEP  379 H= 379:00 +CPU= 20.108
  22:46:45 STEP  380 H= 380:00 +CPU= 19.967
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:48:29 STEP  381 H= 381:00 +CPU= 32.594
  22:48:49 STEP  382 H= 382:00 +CPU= 20.097
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:50:51 STEP  383 H= 383:00 +CPU= 20.210
  22:51:29 STEP  384 H= 384:00 +CPU= 38.148
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000384
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000360
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000372
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000372
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_15.zip
Uploading the intermediate file: upload_file_15.zip
  22:51:48 STEP  385 H= 385:00 +CPU= 19.295
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:53:22 STEP  386 H= 386:00 +CPU= 19.664
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:55:05 STEP  387 H= 387:00 +CPU= 31.885
  22:55:27 STEP  388 H= 388:00 +CPU= 21.250
  22:55:46 STEP  389 H= 389:00 +CPU= 19.821
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:57:51 STEP  390 H= 390:00 +CPU= 33.197
  22:58:10 STEP  391 H= 391:00 +CPU= 19.572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:00:02 STEP  392 H= 392:00 +CPU= 21.158
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:01:46 STEP  393 H= 393:00 +CPU= 32.486
  23:02:08 STEP  394 H= 394:00 +CPU= 21.148
  23:02:28 STEP  395 H= 395:00 +CPU= 20.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:04:55 STEP  396 H= 396:00 +CPU= 35.759
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000396
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:06:26 STEP  397 H= 397:00 +CPU= 20.290
  23:06:48 STEP  398 H= 398:00 +CPU= 21.961
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:08:41 STEP  399 H= 399:00 +CPU= 31.764
  23:09:02 STEP  400 H= 400:00 +CPU= 21.086
  23:09:22 STEP  401 H= 401:00 +CPU= 20.133
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:11:16 STEP  402 H= 402:00 +CPU= 32.188
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:12:58 STEP  403 H= 403:00 +CPU= 20.870
  23:13:19 STEP  404 H= 404:00 +CPU= 20.856
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:15:01 STEP  405 H= 405:00 +CPU= 31.655
  23:15:22 STEP  406 H= 406:00 +CPU= 20.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:17:14 STEP  407 H= 407:00 +CPU= 20.348
  23:17:51 STEP  408 H= 408:00 +CPU= 37.465
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000408
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000384
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000396
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000396
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_16.zip
Uploading the intermediate file: upload_file_16.zip
  23:18:11 STEP  409 H= 409:00 +CPU= 19.604
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:19:41 STEP  410 H= 410:00 +CPU= 20.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:21:35 STEP  411 H= 411:00 +CPU= 32.061
  23:21:56 STEP  412 H= 412:00 +CPU= 20.854
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:23:37 STEP  413 H= 413:00 +CPU= 19.828
  23:24:09 STEP  414 H= 414:00 +CPU= 32.094
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:32 STEP  415 H= 415:00 +CPU= 20.537
  23:25:53 STEP  416 H= 416:00 +CPU= 20.839
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:27:45 STEP  417 H= 417:00 +CPU= 32.435
  23:28:05 STEP  418 H= 418:00 +CPU= 20.309
  23:28:25 STEP  419 H= 419:00 +CPU= 19.757
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:30:31 STEP  420 H= 420:00 +CPU= 35.370
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000420
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000420
  23:31:52 STEP  421 H= 421:00 +CPU= 20.178
  23:32:12 STEP  422 H= 422:00 +CPU= 20.256
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:34:47 STEP  423 H= 423:00 +CPU= 32.395
  23:35:08 STEP  424 H= 424:00 +CPU= 20.766
  23:35:28 STEP  425 H= 425:00 +CPU= 19.910
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:37:22 STEP  426 H= 426:00 +CPU= 32.716
  23:37:41 STEP  427 H= 427:00 +CPU= 19.918
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:39:15 STEP  428 H= 428:00 +CPU= 20.983
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:41:07 STEP  429 H= 429:00 +CPU= 31.548
  23:41:28 STEP  430 H= 430:00 +CPU= 21.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:42:50 STEP  431 H= 431:00 +CPU= 20.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:44:48 STEP  432 H= 432:00 +CPU= 36.638
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000432
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000408
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000420
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000420
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_17.zip
Uploading the intermediate file: upload_file_17.zip
  23:45:09 STEP  433 H= 433:00 +CPU= 20.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:46:21 STEP  434 H= 434:00 +CPU= 20.692
  23:46:52 STEP  435 H= 435:00 +CPU= 31.589
  23:47:13 STEP  436 H= 436:00 +CPU= 20.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:49:24 STEP  437 H= 437:00 +CPU= 20.664
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:50:58 STEP  438 H= 438:00 +CPU= 32.756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:52:00 STEP  439 H= 439:00 +CPU= 20.456
  23:52:21 STEP  440 H= 440:00 +CPU= 20.662
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:04 STEP  441 H= 441:00 +CPU= 31.937
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0611
(argv4) batchid: 965
(argv5) wuid: 12182255
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e4de9a12f692fddd738cc621765c0007 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0611_1996050100_123_965_12182255.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0611_1996050100_123_965_12182255.zip
ic_ancil_file: ic_ancil_12182255
ifsdata_file: ifsdata_12182255
climate_data_file: clim_data_12182255
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dffdd5bebc1f978cf06cbfc508d2c0bb to: /var/lib/boinc-client/slots/9/ic_ancil_12182255.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12182255.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a444f6d310b278d4e8f286702082644e to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12182255.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12182255.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12182255.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12182255.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12182255.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12182255.xml
-- Model is restarting --
Adjusting last_iter, 442, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12182255.xml
last_cpu_time: 15843
upload_file_number: 18
last_iter: 431
last_upload: 1555200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017
The child process has been launched with process id: 91367
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:91367:91367] [20230119:235540:1674140140:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:57:19 STEP  432 H= 432:00 +CPU= 35.646
  23:57:39 STEP  433 H= 433:00 +CPU= 19.907
  23:57:59 STEP  434 H= 434:00 +CPU= 19.631
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:59:52 STEP  435 H= 435:00 +CPU= 32.053
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:01:15 STEP  436 H= 436:00 +CPU= 20.820
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:02:35 STEP  437 H= 437:00 +CPU= 20.291
  00:03:08 STEP  438 H= 438:00 +CPU= 32.261
  00:03:27 STEP  439 H= 439:00 +CPU= 19.690
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:05:18 STEP  440 H= 440:00 +CPU= 20.810
  00:05:49 STEP  441 H= 441:00 +CPU= 30.620
  00:06:09 STEP  442 H= 442:00 +CPU= 19.926
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:08:02 STEP  443 H= 443:00 +CPU= 20.670
  00:08:37 STEP  444 H= 444:00 +CPU= 34.789
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000444
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:10:08 STEP  445 H= 445:00 +CPU= 20.879
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:11:40 STEP  446 H= 446:00 +CPU= 21.022
  00:12:12 STEP  447 H= 447:00 +CPU= 32.628
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:13:55 STEP  448 H= 448:00 +CPU= 20.966
  00:14:16 STEP  449 H= 449:00 +CPU= 21.197
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:16:10 STEP  450 H= 450:00 +CPU= 32.992
  00:16:30 STEP  451 H= 451:00 +CPU= 19.716
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:18:01 STEP  452 H= 452:00 +CPU= 20.346
  00:18:33 STEP  453 H= 453:00 +CPU= 31.890
  00:18:53 STEP  454 H= 454:00 +CPU= 19.893
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:20:35 STEP  455 H= 455:00 +CPU= 21.765
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:22:25 STEP  456 H= 456:00 +CPU= 37.470
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_18.zip
Uploading the intermediate file: upload_file_18.zip
  00:22:45 STEP  457 H= 457:00 +CPU= 20.225
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:24:21 STEP  458 H= 458:00 +CPU= 20.123
  00:24:53 STEP  459 H= 459:00 +CPU= 32.072
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:26:44 STEP  460 H= 460:00 +CPU= 20.155
  00:27:06 STEP  461 H= 461:00 +CPU= 21.301
  00:27:37 STEP  462 H= 462:00 +CPU= 31.533
  00:27:56 STEP  463 H= 463:00 +CPU= 18.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:29:38 STEP  464 H= 464:00 +CPU= 20.701
  00:30:10 STEP  465 H= 465:00 +CPU= 31.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:32:02 STEP  466 H= 466:00 +CPU= 21.291
  00:32:22 STEP  467 H= 467:00 +CPU= 19.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:34:08 STEP  468 H= 468:00 +CPU= 34.953
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000468
  00:34:28 STEP  469 H= 469:00 +CPU= 19.871
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:36:11 STEP  470 H= 470:00 +CPU= 21.065
  00:36:43 STEP  471 H= 471:00 +CPU= 31.796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:38:05 STEP  472 H= 472:00 +CPU= 20.542
  00:38:25 STEP  473 H= 473:00 +CPU= 20.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:39:58 STEP  474 H= 474:00 +CPU= 32.634
  00:40:18 STEP  475 H= 475:00 +CPU= 19.646
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:41:49 STEP  476 H= 476:00 +CPU= 20.832
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:43:21 STEP  477 H= 477:00 +CPU= 32.216
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:44:53 STEP  478 H= 478:00 +CPU= 20.873
  00:45:15 STEP  479 H= 479:00 +CPU= 21.517
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:47:04 STEP  480 H= 480:00 +CPU= 37.888
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_19.zip
Uploading the intermediate file: upload_file_19.zip
  00:47:23 STEP  481 H= 481:00 +CPU= 19.908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:49:05 STEP  482 H= 482:00 +CPU= 20.282
  00:49:37 STEP  483 H= 483:00 +CPU= 31.598
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:51:10 STEP  484 H= 484:00 +CPU= 21.016
  00:51:30 STEP  485 H= 485:00 +CPU= 20.362
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:53:03 STEP  486 H= 486:00 +CPU= 32.120
  00:53:23 STEP  487 H= 487:00 +CPU= 20.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:55:05 STEP  488 H= 488:00 +CPU= 21.416
  00:55:37 STEP  489 H= 489:00 +CPU= 31.688
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:57:48 STEP  490 H= 490:00 +CPU= 20.668
  00:58:08 STEP  491 H= 491:00 +CPU= 20.045
  00:58:42 STEP  492 H= 492:00 +CPU= 34.224
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000492
  00:59:02 STEP  493 H= 493:00 +CPU= 19.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:01:05 STEP  494 H= 494:00 +CPU= 21.187
  01:01:37 STEP  495 H= 495:00 +CPU= 31.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:03:17 STEP  496 H= 496:00 +CPU= 20.393
  01:03:38 STEP  497 H= 497:00 +CPU= 20.836
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:05:21 STEP  498 H= 498:00 +CPU= 31.934
  01:05:41 STEP  499 H= 499:00 +CPU= 19.846
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:07:04 STEP  500 H= 500:00 +CPU= 20.723
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:08:37 STEP  501 H= 501:00 +CPU= 31.962
  01:08:58 STEP  502 H= 502:00 +CPU= 21.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:10:21 STEP  503 H= 503:00 +CPU= 20.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:11:50 STEP  504 H= 504:00 +CPU= 37.858
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_20.zip
Uploading the intermediate file: upload_file_20.zip
  01:12:10 STEP  505 H= 505:00 +CPU= 19.683
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:13:38 STEP  506 H= 506:00 +CPU= 19.543
  01:14:10 STEP  507 H= 507:00 +CPU= 31.929
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:15:43 STEP  508 H= 508:00 +CPU= 21.138
  01:16:04 STEP  509 H= 509:00 +CPU= 21.285
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:17:47 STEP  510 H= 510:00 +CPU= 32.091
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:18:58 STEP  511 H= 511:00 +CPU= 19.849
  01:19:19 STEP  512 H= 512:00 +CPU= 20.256
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:21:03 STEP  513 H= 513:00 +CPU= 31.673
  01:21:23 STEP  514 H= 514:00 +CPU= 20.025
  01:21:43 STEP  515 H= 515:00 +CPU= 20.260
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:23:19 STEP  516 H= 516:00 +CPU= 35.815
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:24:40 STEP  517 H= 517:00 +CPU= 20.059
  01:25:02 STEP  518 H= 518:00 +CPU= 21.830
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:27:05 STEP  519 H= 519:00 +CPU= 32.352
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:28:17 STEP  520 H= 520:00 +CPU= 20.852
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:29:29 STEP  521 H= 521:00 +CPU= 20.406
  01:30:01 STEP  522 H= 522:00 +CPU= 31.897
  01:30:21 STEP  523 H= 523:00 +CPU= 20.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:32:05 STEP  524 H= 524:00 +CPU= 21.392
  01:32:37 STEP  525 H= 525:00 +CPU= 31.969
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:34:08 STEP  526 H= 526:00 +CPU= 21.168
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:35:31 STEP  527 H= 527:00 +CPU= 20.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:37:09 STEP  528 H= 528:00 +CPU= 37.911
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_21.zip
Uploading the intermediate file: upload_file_21.zip
  01:37:29 STEP  529 H= 529:00 +CPU= 20.739
  01:37:52 STEP  530 H= 530:00 +CPU= 22.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:39:58 STEP  531 H= 531:00 +CPU= 34.493
  01:40:20 STEP  532 H= 532:00 +CPU= 21.539
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:45:11 STEP  533 H= 533:00 +CPU= 21.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:52:47 STEP  534 H= 534:00 +CPU= 33.996
  01:53:07 STEP  535 H= 535:00 +CPU= 20.562
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:54:50 STEP  536 H= 536:00 +CPU= 21.391
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:56:43 STEP  537 H= 537:00 +CPU= 32.596
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1996050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0611
(argv4) batchid: 965
(argv5) wuid: 12182255
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e4de9a12f692fddd738cc621765c0007 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0611_1996050100_123_965_12182255.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0611_1996050100_123_965_12182255.zip
ic_ancil_file: ic_ancil_12182255
ifsdata_file: ifsdata_12182255
climate_data_file: clim_data_12182255
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_dffdd5bebc1f978cf06cbfc508d2c0bb to: /var/lib/boinc-client/slots/9/ic_ancil_12182255.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12182255.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_a444f6d310b278d4e8f286702082644e to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12182255.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12182255.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12182255.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12182255.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12182255.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12182255.xml
-- Model is restarting --
Adjusting last_iter, 538, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12182255.xml
last_cpu_time: 18480
upload_file_number: 22
last_iter: 527
last_upload: 1900800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017
The child process has been launched with process id: 111153
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:111153:111153] [20230120:020004:1674147604:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:01:54 STEP  528 H= 528:00 +CPU= 37.406
  02:02:14 STEP  529 H= 529:00 +CPU= 19.783
  02:02:35 STEP  530 H= 530:00 +CPU= 20.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:04:40 STEP  531 H= 531:00 +CPU= 33.344
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:06:33 STEP  532 H= 532:00 +CPU= 21.530
  02:06:55 STEP  533 H= 533:00 +CPU= 21.384
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:08:39 STEP  534 H= 534:00 +CPU= 34.266
  02:09:00 STEP  535 H= 535:00 +CPU= 20.927
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:11:13 STEP  536 H= 536:00 +CPU= 21.580
  02:11:48 STEP  537 H= 537:00 +CPU= 34.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:13:50 STEP  538 H= 538:00 +CPU= 21.423
  02:14:10 STEP  539 H= 539:00 +CPU= 20.163
  02:14:45 STEP  540 H= 540:00 +CPU= 35.381
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000540
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:16:47 STEP  541 H= 541:00 +CPU= 20.248
  02:17:08 STEP  542 H= 542:00 +CPU= 20.969
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:19:32 STEP  543 H= 543:00 +CPU= 33.079
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:21:55 STEP  544 H= 544:00 +CPU= 21.713
  02:22:17 STEP  545 H= 545:00 +CPU= 21.422
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:24:23 STEP  546 H= 546:00 +CPU= 34.503
  02:24:44 STEP  547 H= 547:00 +CPU= 20.837
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:26:48 STEP  548 H= 548:00 +CPU= 21.875
  02:27:22 STEP  549 H= 549:00 +CPU= 33.111
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:29:55 STEP  550 H= 550:00 +CPU= 21.689
  02:30:16 STEP  551 H= 551:00 +CPU= 21.592
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:33:27 STEP  552 H= 552:00 +CPU= 39.069
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12182255/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0611_1996050100_123_965_12182255_0_r942371017_22.zip
Uploading the intermediate file: upload_file_22.zip
  02:33:49 STEP  553 H= 553:00 +CPU= 21.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:35:42 STEP  554 H= 554:00 +CPU= 21.861
  02:36:15 STEP  555 H= 555:00 +CPU= 32.490
  02:36:36 STEP  556 H= 556:00 +CPU= 21.202
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:38:40 STEP  557 H= 557:00 +CPU= 21.672
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:41:15 STEP  558 H= 558:00 +CPU= 33.442
  02:41:36 STEP  559 H= 559:00 +CPU= 21.301
  02:41:57 STEP  560 H= 560:00 +CPU= 20.894
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:44:21 STEP  561 H= 561:00 +CPU= 32.554
  02:44:42 STEP  562 H= 562:00 +CPU= 20.988
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:46:44 STEP  563 H= 563:00 +CPU= 20.435
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net