climateprediction.net home page
Task 22267713

Task 22267713

Name oifs_43r3_ps_0862_1992050100_123_961_12178506_1
Workunit 12178506
Created 22 Dec 2022, 12:20:03 UTC
Sent 12 Jan 2023, 2:50:21 UTC
Report deadline 11 Feb 2023, 2:50:21 UTC
Received 13 Jan 2023, 0:09:34 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532385
Run time 17 hours 57 min 58 sec
CPU time 17 hours 44 min 22 sec
Validate state Invalid
Credit 11,007.36
Device peak FLOPS 5.61 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,797.49 MB
Peak swap size 4,978.17 MB
Peak disk usage 7,180.16 MB
Stderr 
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
CMGGhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_99.zip
Uploading the intermediate file: upload_file_99.zip
  20:06:53 STEP 2401 H=2401:00 +CPU= 22.580
  20:07:15 STEP 2402 H=2402:00 +CPU= 21.938
  20:07:49 STEP 2403 H=2403:00 +CPU= 33.884
  20:08:10 STEP 2404 H=2404:00 +CPU= 21.272
  20:08:32 STEP 2405 H=2405:00 +CPU= 21.370
  20:09:08 STEP 2406 H=2406:00 +CPU= 35.896
  20:09:30 STEP 2407 H=2407:00 +CPU= 21.867
  20:09:54 STEP 2408 H=2408:00 +CPU= 24.409
  20:10:30 STEP 2409 H=2409:00 +CPU= 36.057
  20:10:52 STEP 2410 H=2410:00 +CPU= 21.673
  20:11:13 STEP 2411 H=2411:00 +CPU= 20.925
  20:11:52 STEP 2412 H=2412:00 +CPU= 38.837
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002412
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002412
  20:12:13 STEP 2413 H=2413:00 +CPU= 20.978
  20:12:35 STEP 2414 H=2414:00 +CPU= 21.775
  20:13:09 STEP 2415 H=2415:00 +CPU= 34.321
  20:13:31 STEP 2416 H=2416:00 +CPU= 21.541
  20:13:52 STEP 2417 H=2417:00 +CPU= 20.865
  20:14:26 STEP 2418 H=2418:00 +CPU= 34.556
  20:14:47 STEP 2419 H=2419:00 +CPU= 20.834
  20:15:09 STEP 2420 H=2420:00 +CPU= 21.209
  20:15:42 STEP 2421 H=2421:00 +CPU= 33.719
  20:16:04 STEP 2422 H=2422:00 +CPU= 21.459
  20:16:25 STEP 2423 H=2423:00 +CPU= 21.252
  20:17:05 STEP 2424 H=2424:00 +CPU= 39.447
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002424
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_100.zip
Uploading the intermediate file: upload_file_100.zip
  20:17:25 STEP 2425 H=2425:00 +CPU= 20.200
  20:17:46 STEP 2426 H=2426:00 +CPU= 21.008
  20:18:21 STEP 2427 H=2427:00 +CPU= 35.620
  20:18:43 STEP 2428 H=2428:00 +CPU= 21.800
  20:19:04 STEP 2429 H=2429:00 +CPU= 20.671
  20:19:40 STEP 2430 H=2430:00 +CPU= 35.839
  20:20:01 STEP 2431 H=2431:00 +CPU= 21.492
  20:20:24 STEP 2432 H=2432:00 +CPU= 22.305
  20:20:58 STEP 2433 H=2433:00 +CPU= 34.622
  20:21:20 STEP 2434 H=2434:00 +CPU= 21.317
  20:21:41 STEP 2435 H=2435:00 +CPU= 21.042
  20:22:20 STEP 2436 H=2436:00 +CPU= 38.758
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002436
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002436
  20:22:41 STEP 2437 H=2437:00 +CPU= 20.938
  20:23:04 STEP 2438 H=2438:00 +CPU= 23.195
  20:23:39 STEP 2439 H=2439:00 +CPU= 35.463
  20:24:01 STEP 2440 H=2440:00 +CPU= 21.904
  20:24:23 STEP 2441 H=2441:00 +CPU= 21.113
  20:24:58 STEP 2442 H=2442:00 +CPU= 35.507
  20:25:21 STEP 2443 H=2443:00 +CPU= 22.830
  20:25:43 STEP 2444 H=2444:00 +CPU= 21.435
  20:26:16 STEP 2445 H=2445:00 +CPU= 33.439
  20:26:39 STEP 2446 H=2446:00 +CPU= 22.513
  20:27:04 STEP 2447 H=2447:00 +CPU= 25.327
  20:27:47 STEP 2448 H=2448:00 +CPU= 43.203
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002448
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_101.zip
Uploading the intermediate file: upload_file_101.zip
  20:28:09 STEP 2449 H=2449:00 +CPU= 21.588
  20:28:32 STEP 2450 H=2450:00 +CPU= 23.171
  20:29:08 STEP 2451 H=2451:00 +CPU= 35.230
  20:29:29 STEP 2452 H=2452:00 +CPU= 21.489
  20:29:52 STEP 2453 H=2453:00 +CPU= 22.486
  20:30:27 STEP 2454 H=2454:00 +CPU= 35.233
  20:30:48 STEP 2455 H=2455:00 +CPU= 21.242
  20:31:11 STEP 2456 H=2456:00 +CPU= 23.112
  20:31:48 STEP 2457 H=2457:00 +CPU= 36.706
  20:32:11 STEP 2458 H=2458:00 +CPU= 22.744
  20:32:31 STEP 2459 H=2459:00 +CPU= 20.047
  20:33:08 STEP 2460 H=2460:00 +CPU= 36.644
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002460
  20:33:30 STEP 2461 H=2461:00 +CPU= 21.783
  20:33:51 STEP 2462 H=2462:00 +CPU= 21.358
  20:34:25 STEP 2463 H=2463:00 +CPU= 33.873
  20:34:46 STEP 2464 H=2464:00 +CPU= 21.389
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0862
(argv4) batchid: 961
(argv5) wuid: 12178506
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb610f9379f7a4c996cb9ee03342bc7a to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0862_1992050100_123_961_12178506.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0862_1992050100_123_961_12178506.zip
ic_ancil_file: ic_ancil_12178506
ifsdata_file: ifsdata_12178506
climate_data_file: clim_data_12178506
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f534cf323e204d073a4e655daf08a907 to: /var/lib/boinc-client/slots/11/ic_ancil_12178506.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12178506.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75ba7eaa749de6be0ef9c42aa090a1c3 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12178506.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12178506.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12178506.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12178506.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12178506.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12178506.xml
-- Model is restarting --
Adjusting last_iter, 2464, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12178506.xml
last_cpu_time: 55793
upload_file_number: 102
last_iter: 2447
last_upload: 8812800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854
The child process has been launched with process id: 738699
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:738699:738699] [20230112:205346:1673556826:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:54:26 STEP 2448 H=2448:00 +CPU= 37.801
  20:54:47 STEP 2449 H=2449:00 +CPU= 20.709
  20:55:08 STEP 2450 H=2450:00 +CPU= 21.143
  20:55:42 STEP 2451 H=2451:00 +CPU= 33.633
  20:56:04 STEP 2452 H=2452:00 +CPU= 22.008
  20:56:24 STEP 2453 H=2453:00 +CPU= 20.216
  20:56:59 STEP 2454 H=2454:00 +CPU= 34.254
  20:57:21 STEP 2455 H=2455:00 +CPU= 22.799
  20:57:47 STEP 2456 H=2456:00 +CPU= 25.462
  20:58:26 STEP 2457 H=2457:00 +CPU= 38.900
  20:58:51 STEP 2458 H=2458:00 +CPU= 25.539
  20:59:15 STEP 2459 H=2459:00 +CPU= 23.957
  20:59:56 STEP 2460 H=2460:00 +CPU= 40.937
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002460
  21:00:18 STEP 2461 H=2461:00 +CPU= 21.649
  21:00:40 STEP 2462 H=2462:00 +CPU= 22.409
  21:01:15 STEP 2463 H=2463:00 +CPU= 34.271
  21:01:37 STEP 2464 H=2464:00 +CPU= 21.988
  21:01:58 STEP 2465 H=2465:00 +CPU= 20.734
  21:02:33 STEP 2466 H=2466:00 +CPU= 34.935
  21:02:54 STEP 2467 H=2467:00 +CPU= 21.551
  21:03:16 STEP 2468 H=2468:00 +CPU= 21.403
  21:03:49 STEP 2469 H=2469:00 +CPU= 33.239
  21:04:10 STEP 2470 H=2470:00 +CPU= 21.169
  21:04:32 STEP 2471 H=2471:00 +CPU= 21.995
  21:05:13 STEP 2472 H=2472:00 +CPU= 39.796
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_102.zip
Uploading the intermediate file: upload_file_102.zip
  21:05:34 STEP 2473 H=2473:00 +CPU= 21.206
  21:05:56 STEP 2474 H=2474:00 +CPU= 21.755
  21:06:29 STEP 2475 H=2475:00 +CPU= 33.097
  21:06:50 STEP 2476 H=2476:00 +CPU= 21.071
  21:07:11 STEP 2477 H=2477:00 +CPU= 20.452
  21:07:45 STEP 2478 H=2478:00 +CPU= 33.829
  21:08:05 STEP 2479 H=2479:00 +CPU= 20.193
  21:08:26 STEP 2480 H=2480:00 +CPU= 20.788
  21:08:58 STEP 2481 H=2481:00 +CPU= 32.805
  21:09:20 STEP 2482 H=2482:00 +CPU= 21.123
  21:09:40 STEP 2483 H=2483:00 +CPU= 20.455
  21:10:15 STEP 2484 H=2484:00 +CPU= 34.610
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002484
  21:10:34 STEP 2485 H=2485:00 +CPU= 18.723
  21:10:56 STEP 2486 H=2486:00 +CPU= 21.910
  21:11:29 STEP 2487 H=2487:00 +CPU= 32.889
  21:11:50 STEP 2488 H=2488:00 +CPU= 20.770
  21:12:10 STEP 2489 H=2489:00 +CPU= 20.147
  21:12:44 STEP 2490 H=2490:00 +CPU= 33.661
  21:13:04 STEP 2491 H=2491:00 +CPU= 20.256
  21:13:25 STEP 2492 H=2492:00 +CPU= 20.995
  21:13:58 STEP 2493 H=2493:00 +CPU= 32.926
  21:14:19 STEP 2494 H=2494:00 +CPU= 20.751
  21:14:40 STEP 2495 H=2495:00 +CPU= 21.005
  21:15:21 STEP 2496 H=2496:00 +CPU= 40.237
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_103.zip
Uploading the intermediate file: upload_file_103.zip
  21:15:42 STEP 2497 H=2497:00 +CPU= 21.281
  21:16:04 STEP 2498 H=2498:00 +CPU= 22.035
  21:16:38 STEP 2499 H=2499:00 +CPU= 34.365
  21:17:00 STEP 2500 H=2500:00 +CPU= 21.915
  21:17:21 STEP 2501 H=2501:00 +CPU= 20.992
  21:17:56 STEP 2502 H=2502:00 +CPU= 34.527
  21:18:17 STEP 2503 H=2503:00 +CPU= 21.072
  21:18:39 STEP 2504 H=2504:00 +CPU= 21.061
  21:19:11 STEP 2505 H=2505:00 +CPU= 32.707
  21:19:33 STEP 2506 H=2506:00 +CPU= 21.006
  21:19:53 STEP 2507 H=2507:00 +CPU= 20.200
  21:20:29 STEP 2508 H=2508:00 +CPU= 35.939
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002508
  21:20:49 STEP 2509 H=2509:00 +CPU= 19.817
  21:21:11 STEP 2510 H=2510:00 +CPU= 21.570
  21:21:44 STEP 2511 H=2511:00 +CPU= 33.024
  21:22:04 STEP 2512 H=2512:00 +CPU= 20.568
  21:22:24 STEP 2513 H=2513:00 +CPU= 20.053
  21:22:58 STEP 2514 H=2514:00 +CPU= 33.175
  21:23:19 STEP 2515 H=2515:00 +CPU= 21.629
  21:23:42 STEP 2516 H=2516:00 +CPU= 22.707
  21:24:16 STEP 2517 H=2517:00 +CPU= 33.069
  21:24:37 STEP 2518 H=2518:00 +CPU= 20.583
  21:24:58 STEP 2519 H=2519:00 +CPU= 21.018
  21:25:38 STEP 2520 H=2520:00 +CPU= 38.709
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_104.zip
Uploading the intermediate file: upload_file_104.zip
  21:25:58 STEP 2521 H=2521:00 +CPU= 20.530
  21:26:19 STEP 2522 H=2522:00 +CPU= 20.730
  21:26:53 STEP 2523 H=2523:00 +CPU= 33.710
  21:27:16 STEP 2524 H=2524:00 +CPU= 21.899
  21:27:37 STEP 2525 H=2525:00 +CPU= 20.674
  21:28:12 STEP 2526 H=2526:00 +CPU= 34.512
  21:28:33 STEP 2527 H=2527:00 +CPU= 21.261
  21:28:56 STEP 2528 H=2528:00 +CPU= 22.184
  21:29:30 STEP 2529 H=2529:00 +CPU= 34.378
  21:29:51 STEP 2530 H=2530:00 +CPU= 21.314
  21:30:12 STEP 2531 H=2531:00 +CPU= 20.518
  21:30:48 STEP 2532 H=2532:00 +CPU= 36.303
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002532
  21:31:10 STEP 2533 H=2533:00 +CPU= 21.459
  21:31:34 STEP 2534 H=2534:00 +CPU= 24.165
  21:32:10 STEP 2535 H=2535:00 +CPU= 35.703
  21:32:32 STEP 2536 H=2536:00 +CPU= 21.325
  21:32:52 STEP 2537 H=2537:00 +CPU= 20.193
  21:33:27 STEP 2538 H=2538:00 +CPU= 34.407
  21:33:48 STEP 2539 H=2539:00 +CPU= 21.216
  21:34:09 STEP 2540 H=2540:00 +CPU= 20.772
  21:34:42 STEP 2541 H=2541:00 +CPU= 33.241
  21:35:04 STEP 2542 H=2542:00 +CPU= 21.230
  21:35:25 STEP 2543 H=2543:00 +CPU= 21.273
  21:36:05 STEP 2544 H=2544:00 +CPU= 39.323
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_105.zip
Uploading the intermediate file: upload_file_105.zip
  21:36:27 STEP 2545 H=2545:00 +CPU= 22.673
  21:36:51 STEP 2546 H=2546:00 +CPU= 23.686
  21:37:27 STEP 2547 H=2547:00 +CPU= 35.071
  21:37:48 STEP 2548 H=2548:00 +CPU= 20.671
  21:38:07 STEP 2549 H=2549:00 +CPU= 19.382
  21:38:41 STEP 2550 H=2550:00 +CPU= 33.387
  21:39:02 STEP 2551 H=2551:00 +CPU= 20.927
  21:39:23 STEP 2552 H=2552:00 +CPU= 20.748
  21:39:56 STEP 2553 H=2553:00 +CPU= 32.591
  21:40:17 STEP 2554 H=2554:00 +CPU= 20.550
  21:40:37 STEP 2555 H=2555:00 +CPU= 19.648
  21:41:13 STEP 2556 H=2556:00 +CPU= 36.350
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002556
  21:41:34 STEP 2557 H=2557:00 +CPU= 20.514
  21:41:56 STEP 2558 H=2558:00 +CPU= 21.167
  21:42:28 STEP 2559 H=2559:00 +CPU= 31.775
  21:42:48 STEP 2560 H=2560:00 +CPU= 20.252
  21:43:08 STEP 2561 H=2561:00 +CPU= 19.671
  21:43:43 STEP 2562 H=2562:00 +CPU= 33.760
  21:44:03 STEP 2563 H=2563:00 +CPU= 19.486
  21:44:24 STEP 2564 H=2564:00 +CPU= 20.737
  21:44:58 STEP 2565 H=2565:00 +CPU= 34.198
  21:45:20 STEP 2566 H=2566:00 +CPU= 21.764
  21:45:42 STEP 2567 H=2567:00 +CPU= 21.647
  21:46:23 STEP 2568 H=2568:00 +CPU= 40.388
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_106.zip
Uploading the intermediate file: upload_file_106.zip
  21:46:45 STEP 2569 H=2569:00 +CPU= 21.102
  21:47:05 STEP 2570 H=2570:00 +CPU= 19.820
  21:47:38 STEP 2571 H=2571:00 +CPU= 32.279
  21:48:00 STEP 2572 H=2572:00 +CPU= 21.028
  21:48:20 STEP 2573 H=2573:00 +CPU= 20.167
  21:48:54 STEP 2574 H=2574:00 +CPU= 33.565
  21:49:14 STEP 2575 H=2575:00 +CPU= 20.044
  21:49:35 STEP 2576 H=2576:00 +CPU= 20.696
  21:50:07 STEP 2577 H=2577:00 +CPU= 32.468
  21:50:28 STEP 2578 H=2578:00 +CPU= 20.605
  21:50:48 STEP 2579 H=2579:00 +CPU= 20.504
  21:51:26 STEP 2580 H=2580:00 +CPU= 37.318
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002580
  21:51:45 STEP 2581 H=2581:00 +CPU= 19.102
  21:52:06 STEP 2582 H=2582:00 +CPU= 20.225
  21:52:39 STEP 2583 H=2583:00 +CPU= 33.064
  21:53:00 STEP 2584 H=2584:00 +CPU= 20.393
  21:53:19 STEP 2585 H=2585:00 +CPU= 19.324
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0862
(argv4) batchid: 961
(argv5) wuid: 12178506
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb610f9379f7a4c996cb9ee03342bc7a to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0862_1992050100_123_961_12178506.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0862_1992050100_123_961_12178506.zip
ic_ancil_file: ic_ancil_12178506
ifsdata_file: ifsdata_12178506
climate_data_file: clim_data_12178506
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f534cf323e204d073a4e655daf08a907 to: /var/lib/boinc-client/slots/11/ic_ancil_12178506.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12178506.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75ba7eaa749de6be0ef9c42aa090a1c3 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12178506.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12178506.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12178506.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12178506.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12178506.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12178506.xml
-- Model is restarting --
Adjusting last_iter, 2586, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12178506.xml
last_cpu_time: 59365
upload_file_number: 107
last_iter: 2567
last_upload: 9244800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854
The child process has been launched with process id: 739358
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:739358:739358] [20230112:221542:1673561742:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:16:22 STEP 2568 H=2568:00 +CPU= 37.202
  22:16:43 STEP 2569 H=2569:00 +CPU= 20.843
  22:17:05 STEP 2570 H=2570:00 +CPU= 22.341
  22:17:38 STEP 2571 H=2571:00 +CPU= 32.947
  22:17:59 STEP 2572 H=2572:00 +CPU= 21.111
  22:18:20 STEP 2573 H=2573:00 +CPU= 21.045
  22:18:57 STEP 2574 H=2574:00 +CPU= 36.334
  22:19:18 STEP 2575 H=2575:00 +CPU= 21.523
  22:19:39 STEP 2576 H=2576:00 +CPU= 21.108
  22:20:12 STEP 2577 H=2577:00 +CPU= 32.861
  22:20:33 STEP 2578 H=2578:00 +CPU= 20.756
  22:20:53 STEP 2579 H=2579:00 +CPU= 20.221
  22:21:31 STEP 2580 H=2580:00 +CPU= 37.198
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002580
  22:21:51 STEP 2581 H=2581:00 +CPU= 20.667
  22:22:13 STEP 2582 H=2582:00 +CPU= 21.818
  22:22:47 STEP 2583 H=2583:00 +CPU= 33.786
  22:23:10 STEP 2584 H=2584:00 +CPU= 22.705
  22:23:32 STEP 2585 H=2585:00 +CPU= 22.098
  22:24:06 STEP 2586 H=2586:00 +CPU= 34.169
  22:24:26 STEP 2587 H=2587:00 +CPU= 19.806
  22:24:47 STEP 2588 H=2588:00 +CPU= 20.890
  22:25:21 STEP 2589 H=2589:00 +CPU= 33.493
  22:25:43 STEP 2590 H=2590:00 +CPU= 22.080
  22:26:06 STEP 2591 H=2591:00 +CPU= 23.510
  22:26:45 STEP 2592 H=2592:00 +CPU= 38.942
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_107.zip
Uploading the intermediate file: upload_file_107.zip
  22:27:06 STEP 2593 H=2593:00 +CPU= 20.112
  22:27:27 STEP 2594 H=2594:00 +CPU= 21.135
  22:28:00 STEP 2595 H=2595:00 +CPU= 33.207
  22:28:22 STEP 2596 H=2596:00 +CPU= 21.766
  22:28:44 STEP 2597 H=2597:00 +CPU= 21.895
  22:29:18 STEP 2598 H=2598:00 +CPU= 34.344
  22:29:40 STEP 2599 H=2599:00 +CPU= 21.534
  22:30:02 STEP 2600 H=2600:00 +CPU= 22.303
  22:30:38 STEP 2601 H=2601:00 +CPU= 36.186
  22:31:01 STEP 2602 H=2602:00 +CPU= 22.040
  22:31:21 STEP 2603 H=2603:00 +CPU= 20.465
  22:31:58 STEP 2604 H=2604:00 +CPU= 37.262
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002604
  22:32:19 STEP 2605 H=2605:00 +CPU= 20.641
  22:32:40 STEP 2606 H=2606:00 +CPU= 21.229
  22:33:15 STEP 2607 H=2607:00 +CPU= 34.573
  22:33:37 STEP 2608 H=2608:00 +CPU= 22.251
  22:33:58 STEP 2609 H=2609:00 +CPU= 21.193
  22:34:36 STEP 2610 H=2610:00 +CPU= 37.282
  22:34:58 STEP 2611 H=2611:00 +CPU= 22.083
  22:35:21 STEP 2612 H=2612:00 +CPU= 23.233
  22:35:56 STEP 2613 H=2613:00 +CPU= 34.993
  22:36:16 STEP 2614 H=2614:00 +CPU= 20.223
  22:36:36 STEP 2615 H=2615:00 +CPU= 20.259
  22:37:14 STEP 2616 H=2616:00 +CPU= 37.538
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_108.zip
Uploading the intermediate file: upload_file_108.zip
  22:37:33 STEP 2617 H=2617:00 +CPU= 19.423
  22:37:54 STEP 2618 H=2618:00 +CPU= 20.839
  22:38:29 STEP 2619 H=2619:00 +CPU= 34.898
  22:38:51 STEP 2620 H=2620:00 +CPU= 21.547
  22:39:12 STEP 2621 H=2621:00 +CPU= 20.869
  22:39:47 STEP 2622 H=2622:00 +CPU= 35.320
  22:40:08 STEP 2623 H=2623:00 +CPU= 20.682
  22:40:29 STEP 2624 H=2624:00 +CPU= 21.118
  22:41:03 STEP 2625 H=2625:00 +CPU= 33.834
  22:41:25 STEP 2626 H=2626:00 +CPU= 22.470
  22:41:47 STEP 2627 H=2627:00 +CPU= 22.019
  22:42:26 STEP 2628 H=2628:00 +CPU= 38.056
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002628
  22:42:46 STEP 2629 H=2629:00 +CPU= 20.781
  22:43:08 STEP 2630 H=2630:00 +CPU= 21.459
  22:43:42 STEP 2631 H=2631:00 +CPU= 34.289
  22:44:04 STEP 2632 H=2632:00 +CPU= 21.132
  22:44:23 STEP 2633 H=2633:00 +CPU= 19.657
  22:44:57 STEP 2634 H=2634:00 +CPU= 33.852
  22:45:18 STEP 2635 H=2635:00 +CPU= 20.745
  22:45:41 STEP 2636 H=2636:00 +CPU= 23.036
  22:46:17 STEP 2637 H=2637:00 +CPU= 36.515
  22:46:39 STEP 2638 H=2638:00 +CPU= 21.751
  22:47:00 STEP 2639 H=2639:00 +CPU= 20.828
  22:47:41 STEP 2640 H=2640:00 +CPU= 40.340
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_109.zip
Uploading the intermediate file: upload_file_109.zip
  22:48:03 STEP 2641 H=2641:00 +CPU= 22.258
  22:48:24 STEP 2642 H=2642:00 +CPU= 21.530
  22:48:58 STEP 2643 H=2643:00 +CPU= 33.389
  22:49:19 STEP 2644 H=2644:00 +CPU= 21.258
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           22  MAX ETADOT VERTICAL VEL.=   2.256466379445589E-007
  LON  =    159.750000000000       degrees
  LAT  =   -25.2335138363243       degrees
  22:49:39 STEP 2645 H=2645:00 +CPU= 20.354
  22:50:14 STEP 2646 H=2646:00 +CPU= 34.095
  22:50:35 STEP 2647 H=2647:00 +CPU= 20.748
  22:50:58 STEP 2648 H=2648:00 +CPU= 23.920
  22:51:34 STEP 2649 H=2649:00 +CPU= 35.851
  22:51:56 STEP 2650 H=2650:00 +CPU= 21.477
  22:52:16 STEP 2651 H=2651:00 +CPU= 20.497
  22:52:54 STEP 2652 H=2652:00 +CPU= 37.146
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002652
  22:53:14 STEP 2653 H=2653:00 +CPU= 20.789
  22:53:36 STEP 2654 H=2654:00 +CPU= 21.137
  22:54:09 STEP 2655 H=2655:00 +CPU= 33.259
  22:54:30 STEP 2656 H=2656:00 +CPU= 21.429
  22:54:51 STEP 2657 H=2657:00 +CPU= 20.794
  22:55:25 STEP 2658 H=2658:00 +CPU= 34.416
  22:55:47 STEP 2659 H=2659:00 +CPU= 21.061
  22:56:08 STEP 2660 H=2660:00 +CPU= 21.633
  22:56:42 STEP 2661 H=2661:00 +CPU= 33.617
  22:57:04 STEP 2662 H=2662:00 +CPU= 22.038
  22:57:26 STEP 2663 H=2663:00 +CPU= 21.721
  22:58:06 STEP 2664 H=2664:00 +CPU= 39.879
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_110.zip
Uploading the intermediate file: upload_file_110.zip
  22:58:27 STEP 2665 H=2665:00 +CPU= 21.567
  22:58:50 STEP 2666 H=2666:00 +CPU= 22.598
  22:59:23 STEP 2667 H=2667:00 +CPU= 32.648
  22:59:44 STEP 2668 H=2668:00 +CPU= 21.161
  23:00:06 STEP 2669 H=2669:00 +CPU= 21.472
  23:00:41 STEP 2670 H=2670:00 +CPU= 35.638
  23:01:03 STEP 2671 H=2671:00 +CPU= 22.073
  23:01:25 STEP 2672 H=2672:00 +CPU= 21.432
  23:01:59 STEP 2673 H=2673:00 +CPU= 34.445
  23:02:21 STEP 2674 H=2674:00 +CPU= 21.476
  23:02:42 STEP 2675 H=2675:00 +CPU= 20.658
  23:03:23 STEP 2676 H=2676:00 +CPU= 41.074
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002676
  23:03:44 STEP 2677 H=2677:00 +CPU= 21.571
  23:04:06 STEP 2678 H=2678:00 +CPU= 21.899
  23:04:41 STEP 2679 H=2679:00 +CPU= 34.706
  23:05:03 STEP 2680 H=2680:00 +CPU= 22.138
  23:05:26 STEP 2681 H=2681:00 +CPU= 23.028
  23:06:01 STEP 2682 H=2682:00 +CPU= 33.989
  23:06:21 STEP 2683 H=2683:00 +CPU= 20.074
  23:06:42 STEP 2684 H=2684:00 +CPU= 20.737
  23:07:15 STEP 2685 H=2685:00 +CPU= 33.454
  23:07:39 STEP 2686 H=2686:00 +CPU= 23.844
  23:08:01 STEP 2687 H=2687:00 +CPU= 21.364
  23:08:40 STEP 2688 H=2688:00 +CPU= 39.155
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_111.zip
Uploading the intermediate file: upload_file_111.zip
  23:09:01 STEP 2689 H=2689:00 +CPU= 20.952
  23:09:24 STEP 2690 H=2690:00 +CPU= 22.435
  23:09:58 STEP 2691 H=2691:00 +CPU= 34.694
  23:10:19 STEP 2692 H=2692:00 +CPU= 20.595
  23:10:41 STEP 2693 H=2693:00 +CPU= 21.408
  23:11:16 STEP 2694 H=2694:00 +CPU= 35.231
  23:11:37 STEP 2695 H=2695:00 +CPU= 21.152
  23:11:59 STEP 2696 H=2696:00 +CPU= 21.433
  23:12:32 STEP 2697 H=2697:00 +CPU= 33.262
  23:12:54 STEP 2698 H=2698:00 +CPU= 22.252
  23:13:16 STEP 2699 H=2699:00 +CPU= 21.618
  23:13:54 STEP 2700 H=2700:00 +CPU= 38.116
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002700
  23:14:15 STEP 2701 H=2701:00 +CPU= 21.276
  23:14:37 STEP 2702 H=2702:00 +CPU= 21.670
  23:15:13 STEP 2703 H=2703:00 +CPU= 35.431
  23:15:36 STEP 2704 H=2704:00 +CPU= 23.380
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1992050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0862
(argv4) batchid: 961
(argv5) wuid: 12178506
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/11
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb610f9379f7a4c996cb9ee03342bc7a to: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0862_1992050100_123_961_12178506.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/11/oifs_43r3_ps_0862_1992050100_123_961_12178506.zip
ic_ancil_file: ic_ancil_12178506
ifsdata_file: ifsdata_12178506
climate_data_file: clim_data_12178506
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f534cf323e204d073a4e655daf08a907 to: /var/lib/boinc-client/slots/11/ic_ancil_12178506.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/11/ic_ancil_12178506.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_75ba7eaa749de6be0ef9c42aa090a1c3 to: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12178506.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/11/ifsdata/ifsdata_12178506.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/11/159l_2/clim_data_12178506.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/11/159l_2/clim_data_12178506.zip
Checking for progress XML file: /var/lib/boinc-client/slots/11/progress_file_12178506.xml
Opened progress file ok : /var/lib/boinc-client/slots/11/progress_file_12178506.xml
-- Model is restarting --
Adjusting last_iter, 2705, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/11/progress_file_12178506.xml
last_cpu_time: 62955
upload_file_number: 112
last_iter: 2687
last_upload: 9676800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854
The child process has been launched with process id: 740888
Executing the command: /var/lib/boinc-client/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM:1:1:740888:740888] [20230112:234510:1673567110:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:45:52 STEP 2688 H=2688:00 +CPU= 39.663
  23:46:15 STEP 2689 H=2689:00 +CPU= 22.501
  23:46:38 STEP 2690 H=2690:00 +CPU= 22.979
  23:47:14 STEP 2691 H=2691:00 +CPU= 36.945
  23:47:37 STEP 2692 H=2692:00 +CPU= 22.175
  23:47:58 STEP 2693 H=2693:00 +CPU= 21.344
  23:48:33 STEP 2694 H=2694:00 +CPU= 34.747
  23:48:54 STEP 2695 H=2695:00 +CPU= 20.890
  23:49:16 STEP 2696 H=2696:00 +CPU= 22.463
  23:49:50 STEP 2697 H=2697:00 +CPU= 33.588
  23:50:11 STEP 2698 H=2698:00 +CPU= 21.238
  23:50:31 STEP 2699 H=2699:00 +CPU= 20.191
  23:51:08 STEP 2700 H=2700:00 +CPU= 36.646
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002700
  23:51:29 STEP 2701 H=2701:00 +CPU= 21.256
  23:51:52 STEP 2702 H=2702:00 +CPU= 22.130
  23:52:26 STEP 2703 H=2703:00 +CPU= 33.969
  23:52:47 STEP 2704 H=2704:00 +CPU= 21.912
  23:53:08 STEP 2705 H=2705:00 +CPU= 20.897
  23:53:43 STEP 2706 H=2706:00 +CPU= 34.547
  23:54:03 STEP 2707 H=2707:00 +CPU= 20.509
  23:54:25 STEP 2708 H=2708:00 +CPU= 21.238
  23:54:58 STEP 2709 H=2709:00 +CPU= 33.657
  23:55:20 STEP 2710 H=2710:00 +CPU= 21.100
  23:55:41 STEP 2711 H=2711:00 +CPU= 21.381
  23:56:20 STEP 2712 H=2712:00 +CPU= 39.036
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/11/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12178506/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0862_1992050100_123_961_12178506_1_r1698139854_112.zip
Uploading the intermediate file: upload_file_112.zip
  23:56:41 STEP 2713 H=2713:00 +CPU= 21.065
  23:57:02 STEP 2714 H=2714:00 +CPU= 20.719
  23:57:37 STEP 2715 H=2715:00 +CPU= 34.631
  23:57:59 STEP 2716 H=2716:00 +CPU= 22.077
  23:58:20 STEP 2717 H=2717:00 +CPU= 20.967
  23:58:54 STEP 2718 H=2718:00 +CPU= 34.642
  23:59:15 STEP 2719 H=2719:00 +CPU= 20.045
  23:59:36 STEP 2720 H=2720:00 +CPU= 21.539
  00:00:12 STEP 2721 H=2721:00 +CPU= 35.872
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:17:02 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 7,862,400 47,494 0.0060
09 May 2023 16:16:55 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 6,998,400 41,442 0.0059
09 May 2023 16:16:51 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 6,134,400 35,360 0.0058
09 May 2023 16:16:45 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 5,270,400 29,297 0.0056
09 May 2023 16:16:37 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 4,406,400 23,164 0.0053
09 May 2023 16:16:29 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 3,456,000 16,311 0.0047
09 May 2023 16:16:25 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 2,592,000 12,069 0.0047
09 May 2023 16:16:25 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 2,592,000 12,069 0.0047
09 May 2023 16:16:22 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 1,728,000 8,079 0.0047
09 May 2023 16:16:19 1532385 22267713 oifs_43r3_ps_0862_1992050100_123_961_12178506_1 864,000 4,058 0.0047


©2024 cpdn.org