climateprediction.net home page
Task 22272866

Task 22272866

Name oifs_43r3_ps_0796_2001050100_123_970_12187440_0
Workunit 12187440
Created 23 Dec 2022, 1:02:41 UTC
Sent 29 Dec 2022, 18:32:55 UTC
Report deadline 28 Jan 2023, 18:32:55 UTC
Received 31 Dec 2022, 0:40:26 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1463165
Run time 18 hours 54 min
CPU time 18 hours 52 min 21 sec
Validate state Invalid
Credit 10,160.64
Device peak FLOPS 4.74 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,800.25 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,737.40 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
P 2477 H=2477:00 +CPU= 18.919
  12:17:06 STEP 2478 H=2478:00 +CPU= 31.246
  12:17:25 STEP 2479 H=2479:00 +CPU= 18.918
  12:17:45 STEP 2480 H=2480:00 +CPU= 19.555
  12:18:15 STEP 2481 H=2481:00 +CPU= 30.552
  12:18:35 STEP 2482 H=2482:00 +CPU= 19.563
  12:18:54 STEP 2483 H=2483:00 +CPU= 18.932
  12:19:27 STEP 2484 H=2484:00 +CPU= 33.761
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002484
  12:19:46 STEP 2485 H=2485:00 +CPU= 18.939
  12:20:06 STEP 2486 H=2486:00 +CPU= 19.555
  12:20:37 STEP 2487 H=2487:00 +CPU= 30.526
  12:20:56 STEP 2488 H=2488:00 +CPU= 19.532
  12:21:15 STEP 2489 H=2489:00 +CPU= 18.887
  12:21:46 STEP 2490 H=2490:00 +CPU= 31.190
  12:22:05 STEP 2491 H=2491:00 +CPU= 18.873
  12:22:25 STEP 2492 H=2492:00 +CPU= 19.494
  12:22:55 STEP 2493 H=2493:00 +CPU= 30.522
  12:23:15 STEP 2494 H=2494:00 +CPU= 19.501
  12:23:34 STEP 2495 H=2495:00 +CPU= 19.422
  12:24:09 STEP 2496 H=2496:00 +CPU= 35.452
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_103.zip
Uploading the intermediate file: upload_file_103.zip
  12:24:28 STEP 2497 H=2497:00 +CPU= 18.938
  12:24:48 STEP 2498 H=2498:00 +CPU= 19.500
  12:25:18 STEP 2499 H=2499:00 +CPU= 30.535
  12:25:38 STEP 2500 H=2500:00 +CPU= 19.524
  12:25:57 STEP 2501 H=2501:00 +CPU= 18.903
  12:26:28 STEP 2502 H=2502:00 +CPU= 31.236
  12:26:47 STEP 2503 H=2503:00 +CPU= 18.911
  12:27:07 STEP 2504 H=2504:00 +CPU= 19.533
  12:27:37 STEP 2505 H=2505:00 +CPU= 30.534
  12:27:57 STEP 2506 H=2506:00 +CPU= 19.532
  12:28:16 STEP 2507 H=2507:00 +CPU= 18.905
  12:28:49 STEP 2508 H=2508:00 +CPU= 33.728
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002508
  12:29:08 STEP 2509 H=2509:00 +CPU= 18.905
  12:29:28 STEP 2510 H=2510:00 +CPU= 19.521
  12:29:58 STEP 2511 H=2511:00 +CPU= 30.531
  12:30:18 STEP 2512 H=2512:00 +CPU= 19.536
  12:30:37 STEP 2513 H=2513:00 +CPU= 18.907
  12:31:08 STEP 2514 H=2514:00 +CPU= 31.274
  12:31:27 STEP 2515 H=2515:00 +CPU= 18.891
  12:31:46 STEP 2516 H=2516:00 +CPU= 19.517
  12:32:17 STEP 2517 H=2517:00 +CPU= 30.558
  12:32:36 STEP 2518 H=2518:00 +CPU= 19.504
  12:32:56 STEP 2519 H=2519:00 +CPU= 19.427
  12:33:31 STEP 2520 H=2520:00 +CPU= 35.493
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_104.zip
Uploading the intermediate file: upload_file_104.zip
  12:33:50 STEP 2521 H=2521:00 +CPU= 18.941
  12:34:10 STEP 2522 H=2522:00 +CPU= 19.505
  12:34:40 STEP 2523 H=2523:00 +CPU= 30.544
  12:35:00 STEP 2524 H=2524:00 +CPU= 19.527
  12:35:19 STEP 2525 H=2525:00 +CPU= 18.923
  12:35:50 STEP 2526 H=2526:00 +CPU= 31.285
  12:36:09 STEP 2527 H=2527:00 +CPU= 18.927
  12:36:29 STEP 2528 H=2528:00 +CPU= 19.541
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0796
(argv4) batchid: 970
(argv5) wuid: 12187440
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_afc53c81a3ca741537c706373f715a24 to: /var/lib/boinc/slots/1/oifs_43r3_ps_0796_2001050100_123_970_12187440.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0796_2001050100_123_970_12187440.zip
ic_ancil_file: ic_ancil_12187440
ifsdata_file: ifsdata_12187440
climate_data_file: clim_data_12187440
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a89f0741991f6b2c4bc07ced9d3cd0c2 to: /var/lib/boinc/slots/1/ic_ancil_12187440.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12187440.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e55f366fbb72509f39d366e995cdffa8 to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12187440.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12187440.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12187440.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12187440.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12187440.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12187440.xml
-- Model is restarting --
Adjusting last_iter, 2529, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12187440.xml
last_cpu_time: 59888
upload_file_number: 105
last_iter: 2519
last_upload: 9072000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847
The child process has been launched with process id: 5983
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:5983:5983] [20221230:131832:1672431512:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  13:19:07 STEP 2520 H=2520:00 +CPU= 33.896
  13:19:26 STEP 2521 H=2521:00 +CPU= 18.896
  13:19:46 STEP 2522 H=2522:00 +CPU= 19.515
  13:20:16 STEP 2523 H=2523:00 +CPU= 30.570
  13:20:36 STEP 2524 H=2524:00 +CPU= 19.546
  13:20:55 STEP 2525 H=2525:00 +CPU= 18.920
  13:21:26 STEP 2526 H=2526:00 +CPU= 31.282
  13:21:45 STEP 2527 H=2527:00 +CPU= 18.923
  13:22:05 STEP 2528 H=2528:00 +CPU= 19.556
  13:22:35 STEP 2529 H=2529:00 +CPU= 30.574
  13:22:55 STEP 2530 H=2530:00 +CPU= 19.536
  13:23:14 STEP 2531 H=2531:00 +CPU= 18.926
  13:23:47 STEP 2532 H=2532:00 +CPU= 33.789
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002532
  13:24:06 STEP 2533 H=2533:00 +CPU= 18.934
  13:24:26 STEP 2534 H=2534:00 +CPU= 19.555
  13:24:57 STEP 2535 H=2535:00 +CPU= 30.589
  13:25:16 STEP 2536 H=2536:00 +CPU= 19.548
  13:25:35 STEP 2537 H=2537:00 +CPU= 18.924
  13:26:06 STEP 2538 H=2538:00 +CPU= 31.278
  13:26:25 STEP 2539 H=2539:00 +CPU= 18.911
  13:26:45 STEP 2540 H=2540:00 +CPU= 19.501
  13:27:15 STEP 2541 H=2541:00 +CPU= 30.512
  13:27:35 STEP 2542 H=2542:00 +CPU= 19.478
  13:27:54 STEP 2543 H=2543:00 +CPU= 19.385
  13:28:29 STEP 2544 H=2544:00 +CPU= 35.299
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_105.zip
Uploading the intermediate file: upload_file_105.zip
  13:28:48 STEP 2545 H=2545:00 +CPU= 18.937
  13:29:08 STEP 2546 H=2546:00 +CPU= 19.515
  13:29:38 STEP 2547 H=2547:00 +CPU= 30.573
  13:29:58 STEP 2548 H=2548:00 +CPU= 19.513
  13:30:17 STEP 2549 H=2549:00 +CPU= 18.918
  13:30:48 STEP 2550 H=2550:00 +CPU= 31.279
  13:31:07 STEP 2551 H=2551:00 +CPU= 18.938
  13:31:27 STEP 2552 H=2552:00 +CPU= 19.554
  13:31:57 STEP 2553 H=2553:00 +CPU= 30.581
  13:32:17 STEP 2554 H=2554:00 +CPU= 19.548
  13:32:36 STEP 2555 H=2555:00 +CPU= 18.914
  13:33:10 STEP 2556 H=2556:00 +CPU= 33.772
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002556
  13:33:28 STEP 2557 H=2557:00 +CPU= 18.911
  13:33:48 STEP 2558 H=2558:00 +CPU= 19.541
  13:34:19 STEP 2559 H=2559:00 +CPU= 30.550
  13:34:38 STEP 2560 H=2560:00 +CPU= 19.525
  13:34:57 STEP 2561 H=2561:00 +CPU= 18.905
  13:35:28 STEP 2562 H=2562:00 +CPU= 31.265
  13:35:47 STEP 2563 H=2563:00 +CPU= 18.903
  13:36:07 STEP 2564 H=2564:00 +CPU= 19.511
  13:36:37 STEP 2565 H=2565:00 +CPU= 30.564
  13:36:57 STEP 2566 H=2566:00 +CPU= 19.496
  13:37:16 STEP 2567 H=2567:00 +CPU= 19.404
  13:37:52 STEP 2568 H=2568:00 +CPU= 35.441
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_106.zip
Uploading the intermediate file: upload_file_106.zip
  13:38:11 STEP 2569 H=2569:00 +CPU= 18.931
  13:38:30 STEP 2570 H=2570:00 +CPU= 19.512
  13:39:01 STEP 2571 H=2571:00 +CPU= 30.579
  13:39:20 STEP 2572 H=2572:00 +CPU= 19.528
  13:39:39 STEP 2573 H=2573:00 +CPU= 18.898
  13:40:10 STEP 2574 H=2574:00 +CPU= 31.294
  13:40:29 STEP 2575 H=2575:00 +CPU= 18.944
  13:40:49 STEP 2576 H=2576:00 +CPU= 19.568
  13:41:20 STEP 2577 H=2577:00 +CPU= 30.600
  13:41:39 STEP 2578 H=2578:00 +CPU= 19.552
  13:41:58 STEP 2579 H=2579:00 +CPU= 18.939
  13:42:32 STEP 2580 H=2580:00 +CPU= 33.818
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002580
  13:42:51 STEP 2581 H=2581:00 +CPU= 18.935
  13:43:10 STEP 2582 H=2582:00 +CPU= 19.528
  13:43:41 STEP 2583 H=2583:00 +CPU= 30.579
  13:44:00 STEP 2584 H=2584:00 +CPU= 19.541
  13:44:19 STEP 2585 H=2585:00 +CPU= 18.898
  13:44:51 STEP 2586 H=2586:00 +CPU= 31.279
  13:45:10 STEP 2587 H=2587:00 +CPU= 18.884
  13:45:29 STEP 2588 H=2588:00 +CPU= 19.495
  13:46:00 STEP 2589 H=2589:00 +CPU= 30.547
  13:46:19 STEP 2590 H=2590:00 +CPU= 19.481
  13:46:38 STEP 2591 H=2591:00 +CPU= 19.401
  13:47:14 STEP 2592 H=2592:00 +CPU= 35.448
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_107.zip
Uploading the intermediate file: upload_file_107.zip
  13:47:33 STEP 2593 H=2593:00 +CPU= 18.930
  13:47:52 STEP 2594 H=2594:00 +CPU= 19.504
  13:48:23 STEP 2595 H=2595:00 +CPU= 30.568
  13:48:42 STEP 2596 H=2596:00 +CPU= 19.503
  13:49:01 STEP 2597 H=2597:00 +CPU= 18.918
  13:49:33 STEP 2598 H=2598:00 +CPU= 31.302
  13:49:52 STEP 2599 H=2599:00 +CPU= 18.922
  13:50:11 STEP 2600 H=2600:00 +CPU= 19.547
  13:50:42 STEP 2601 H=2601:00 +CPU= 30.583
  13:51:01 STEP 2602 H=2602:00 +CPU= 19.539
  13:51:20 STEP 2603 H=2603:00 +CPU= 18.915
  13:51:54 STEP 2604 H=2604:00 +CPU= 33.758
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002604
  13:52:13 STEP 2605 H=2605:00 +CPU= 18.904
  13:52:32 STEP 2606 H=2606:00 +CPU= 19.518
  13:53:03 STEP 2607 H=2607:00 +CPU= 30.508
  13:53:22 STEP 2608 H=2608:00 +CPU= 19.515
  13:53:41 STEP 2609 H=2609:00 +CPU= 18.880
  13:54:13 STEP 2610 H=2610:00 +CPU= 31.230
  13:54:31 STEP 2611 H=2611:00 +CPU= 18.880
  13:54:51 STEP 2612 H=2612:00 +CPU= 19.501
  13:55:21 STEP 2613 H=2613:00 +CPU= 30.516
  13:55:41 STEP 2614 H=2614:00 +CPU= 19.473
  13:56:00 STEP 2615 H=2615:00 +CPU= 19.414
  13:56:36 STEP 2616 H=2616:00 +CPU= 35.435
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_108.zip
Uploading the intermediate file: upload_file_108.zip
  13:56:55 STEP 2617 H=2617:00 +CPU= 18.908
  13:57:14 STEP 2618 H=2618:00 +CPU= 19.474
  13:57:45 STEP 2619 H=2619:00 +CPU= 30.534
  13:58:04 STEP 2620 H=2620:00 +CPU= 19.481
  13:58:23 STEP 2621 H=2621:00 +CPU= 18.871
  13:58:54 STEP 2622 H=2622:00 +CPU= 31.260
  13:59:13 STEP 2623 H=2623:00 +CPU= 18.896
  13:59:33 STEP 2624 H=2624:00 +CPU= 19.517
  14:00:03 STEP 2625 H=2625:00 +CPU= 30.553
  14:00:23 STEP 2626 H=2626:00 +CPU= 19.527
  14:00:42 STEP 2627 H=2627:00 +CPU= 18.899
  14:01:15 STEP 2628 H=2628:00 +CPU= 33.722
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002628
  14:01:34 STEP 2629 H=2629:00 +CPU= 18.891
  14:01:54 STEP 2630 H=2630:00 +CPU= 19.510
  14:02:24 STEP 2631 H=2631:00 +CPU= 30.511
  14:02:44 STEP 2632 H=2632:00 +CPU= 19.498
  14:03:03 STEP 2633 H=2633:00 +CPU= 18.880
  14:03:34 STEP 2634 H=2634:00 +CPU= 31.212
  14:03:53 STEP 2635 H=2635:00 +CPU= 18.874
  14:04:12 STEP 2636 H=2636:00 +CPU= 19.491
  14:04:43 STEP 2637 H=2637:00 +CPU= 30.512
  14:05:02 STEP 2638 H=2638:00 +CPU= 19.473
  14:05:22 STEP 2639 H=2639:00 +CPU= 19.394
  14:05:57 STEP 2640 H=2640:00 +CPU= 35.429
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_109.zip
Uploading the intermediate file: upload_file_109.zip
  14:06:16 STEP 2641 H=2641:00 +CPU= 18.933
  14:06:36 STEP 2642 H=2642:00 +CPU= 19.482
  14:07:06 STEP 2643 H=2643:00 +CPU= 30.553
  14:07:26 STEP 2644 H=2644:00 +CPU= 19.524
  14:07:45 STEP 2645 H=2645:00 +CPU= 18.899
  14:08:16 STEP 2646 H=2646:00 +CPU= 31.257
  14:08:35 STEP 2647 H=2647:00 +CPU= 18.908
  14:08:54 STEP 2648 H=2648:00 +CPU= 19.534
  14:09:25 STEP 2649 H=2649:00 +CPU= 30.543
  14:09:44 STEP 2650 H=2650:00 +CPU= 19.534
  14:10:03 STEP 2651 H=2651:00 +CPU= 18.929
  14:10:37 STEP 2652 H=2652:00 +CPU= 33.745
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002652
  14:10:56 STEP 2653 H=2653:00 +CPU= 18.918
  14:11:16 STEP 2654 H=2654:00 +CPU= 19.533
  14:11:46 STEP 2655 H=2655:00 +CPU= 30.524
  14:12:06 STEP 2656 H=2656:00 +CPU= 19.504
  14:12:24 STEP 2657 H=2657:00 +CPU= 18.884
  14:12:56 STEP 2658 H=2658:00 +CPU= 31.223
  14:13:15 STEP 2659 H=2659:00 +CPU= 18.874
  14:13:34 STEP 2660 H=2660:00 +CPU= 19.484
  14:14:05 STEP 2661 H=2661:00 +CPU= 30.504
  14:14:24 STEP 2662 H=2662:00 +CPU= 19.473
  14:14:43 STEP 2663 H=2663:00 +CPU= 19.383
  14:15:19 STEP 2664 H=2664:00 +CPU= 35.436
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_110.zip
Uploading the intermediate file: upload_file_110.zip
  14:15:38 STEP 2665 H=2665:00 +CPU= 18.881
  14:15:57 STEP 2666 H=2666:00 +CPU= 19.482
  14:16:28 STEP 2667 H=2667:00 +CPU= 30.520
  14:16:47 STEP 2668 H=2668:00 +CPU= 19.499
  14:17:06 STEP 2669 H=2669:00 +CPU= 18.892
  14:17:37 STEP 2670 H=2670:00 +CPU= 31.240
  14:17:56 STEP 2671 H=2671:00 +CPU= 18.902
  14:18:16 STEP 2672 H=2672:00 +CPU= 19.529
  14:18:46 STEP 2673 H=2673:00 +CPU= 30.541
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0796
(argv4) batchid: 970
(argv5) wuid: 12187440
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_afc53c81a3ca741537c706373f715a24 to: /var/lib/boinc/slots/1/oifs_43r3_ps_0796_2001050100_123_970_12187440.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0796_2001050100_123_970_12187440.zip
ic_ancil_file: ic_ancil_12187440
ifsdata_file: ifsdata_12187440
climate_data_file: clim_data_12187440
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a89f0741991f6b2c4bc07ced9d3cd0c2 to: /var/lib/boinc/slots/1/ic_ancil_12187440.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12187440.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e55f366fbb72509f39d366e995cdffa8 to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12187440.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12187440.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12187440.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12187440.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12187440.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12187440.xml
-- Model is restarting --
Adjusting last_iter, 2674, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12187440.xml
last_cpu_time: 63503
upload_file_number: 111
last_iter: 2663
last_upload: 9590400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847
The child process has been launched with process id: 6466
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6466:6466] [20221230:154311:1672440191:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  15:43:46 STEP 2664 H=2664:00 +CPU= 33.894
  15:44:05 STEP 2665 H=2665:00 +CPU= 18.898
  15:44:25 STEP 2666 H=2666:00 +CPU= 19.507
  15:44:55 STEP 2667 H=2667:00 +CPU= 30.513
  15:45:15 STEP 2668 H=2668:00 +CPU= 19.515
  15:45:34 STEP 2669 H=2669:00 +CPU= 18.901
  15:46:05 STEP 2670 H=2670:00 +CPU= 31.245
  15:46:24 STEP 2671 H=2671:00 +CPU= 18.923
  15:46:43 STEP 2672 H=2672:00 +CPU= 19.549
  15:47:14 STEP 2673 H=2673:00 +CPU= 30.531
  15:47:33 STEP 2674 H=2674:00 +CPU= 19.551
  15:47:52 STEP 2675 H=2675:00 +CPU= 18.931
  15:48:26 STEP 2676 H=2676:00 +CPU= 33.763
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002676
  15:48:45 STEP 2677 H=2677:00 +CPU= 18.943
  15:49:05 STEP 2678 H=2678:00 +CPU= 19.557
  15:49:35 STEP 2679 H=2679:00 +CPU= 30.557
  15:49:55 STEP 2680 H=2680:00 +CPU= 19.549
  15:50:14 STEP 2681 H=2681:00 +CPU= 18.915
  15:50:45 STEP 2682 H=2682:00 +CPU= 31.245
  15:51:04 STEP 2683 H=2683:00 +CPU= 18.903
  15:51:23 STEP 2684 H=2684:00 +CPU= 19.525
  15:51:54 STEP 2685 H=2685:00 +CPU= 30.542
  15:52:13 STEP 2686 H=2686:00 +CPU= 19.502
  15:52:33 STEP 2687 H=2687:00 +CPU= 19.421
  15:53:08 STEP 2688 H=2688:00 +CPU= 35.245
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_111.zip
Uploading the intermediate file: upload_file_111.zip
  15:53:27 STEP 2689 H=2689:00 +CPU= 18.935
  15:53:47 STEP 2690 H=2690:00 +CPU= 19.490
  15:54:17 STEP 2691 H=2691:00 +CPU= 30.480
  15:54:36 STEP 2692 H=2692:00 +CPU= 19.489
  15:54:55 STEP 2693 H=2693:00 +CPU= 18.880
  15:55:27 STEP 2694 H=2694:00 +CPU= 31.196
  15:55:45 STEP 2695 H=2695:00 +CPU= 18.899
  15:56:05 STEP 2696 H=2696:00 +CPU= 19.525
  15:56:36 STEP 2697 H=2697:00 +CPU= 30.505
  15:56:55 STEP 2698 H=2698:00 +CPU= 19.542
  15:57:14 STEP 2699 H=2699:00 +CPU= 18.918
  15:57:48 STEP 2700 H=2700:00 +CPU= 33.714
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002700
  15:58:07 STEP 2701 H=2701:00 +CPU= 18.905
  15:58:26 STEP 2702 H=2702:00 +CPU= 19.527
  15:58:57 STEP 2703 H=2703:00 +CPU= 30.505
  15:59:16 STEP 2704 H=2704:00 +CPU= 19.505
  15:59:35 STEP 2705 H=2705:00 +CPU= 18.905
  16:00:06 STEP 2706 H=2706:00 +CPU= 31.209
  16:00:25 STEP 2707 H=2707:00 +CPU= 18.883
  16:00:45 STEP 2708 H=2708:00 +CPU= 19.496
  16:01:15 STEP 2709 H=2709:00 +CPU= 30.495
  16:01:35 STEP 2710 H=2710:00 +CPU= 19.491
  16:01:54 STEP 2711 H=2711:00 +CPU= 19.406
  16:02:30 STEP 2712 H=2712:00 +CPU= 35.415
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_112.zip
Uploading the intermediate file: upload_file_112.zip
  16:02:48 STEP 2713 H=2713:00 +CPU= 18.905
  16:03:08 STEP 2714 H=2714:00 +CPU= 19.496
  16:03:38 STEP 2715 H=2715:00 +CPU= 30.512
  16:03:58 STEP 2716 H=2716:00 +CPU= 19.515
  16:04:17 STEP 2717 H=2717:00 +CPU= 18.888
  16:04:48 STEP 2718 H=2718:00 +CPU= 31.231
  16:05:07 STEP 2719 H=2719:00 +CPU= 18.910
  16:05:27 STEP 2720 H=2720:00 +CPU= 19.544
  16:05:57 STEP 2721 H=2721:00 +CPU= 30.535
  16:06:17 STEP 2722 H=2722:00 +CPU= 19.535
  16:06:36 STEP 2723 H=2723:00 +CPU= 18.917
  16:07:09 STEP 2724 H=2724:00 +CPU= 33.705
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002724
  16:07:28 STEP 2725 H=2725:00 +CPU= 18.896
  16:07:48 STEP 2726 H=2726:00 +CPU= 19.508
  16:08:18 STEP 2727 H=2727:00 +CPU= 30.482
  16:08:38 STEP 2728 H=2728:00 +CPU= 19.490
  16:08:57 STEP 2729 H=2729:00 +CPU= 18.889
  16:09:28 STEP 2730 H=2730:00 +CPU= 31.202
  16:09:47 STEP 2731 H=2731:00 +CPU= 18.865
  16:10:06 STEP 2732 H=2732:00 +CPU= 19.494
  16:10:37 STEP 2733 H=2733:00 +CPU= 30.490
  16:10:56 STEP 2734 H=2734:00 +CPU= 19.476
  16:11:16 STEP 2735 H=2735:00 +CPU= 19.392
  16:11:51 STEP 2736 H=2736:00 +CPU= 35.371
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_113.zip
Uploading the intermediate file: upload_file_113.zip
  16:12:10 STEP 2737 H=2737:00 +CPU= 18.898
  16:12:29 STEP 2738 H=2738:00 +CPU= 19.455
  16:13:00 STEP 2739 H=2739:00 +CPU= 30.471
  16:13:19 STEP 2740 H=2740:00 +CPU= 19.460
  16:13:38 STEP 2741 H=2741:00 +CPU= 18.859
  16:14:09 STEP 2742 H=2742:00 +CPU= 31.176
  16:14:28 STEP 2743 H=2743:00 +CPU= 18.875
  16:14:48 STEP 2744 H=2744:00 +CPU= 19.505
  16:15:18 STEP 2745 H=2745:00 +CPU= 30.473
  16:15:38 STEP 2746 H=2746:00 +CPU= 19.502
  16:15:56 STEP 2747 H=2747:00 +CPU= 18.881
  16:16:30 STEP 2748 H=2748:00 +CPU= 33.655
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002748
  16:16:49 STEP 2749 H=2749:00 +CPU= 18.877
  16:17:09 STEP 2750 H=2750:00 +CPU= 19.496
  16:17:39 STEP 2751 H=2751:00 +CPU= 30.412
  16:17:58 STEP 2752 H=2752:00 +CPU= 19.471
  16:18:17 STEP 2753 H=2753:00 +CPU= 18.836
  16:18:48 STEP 2754 H=2754:00 +CPU= 31.114
  16:19:07 STEP 2755 H=2755:00 +CPU= 18.854
  16:19:27 STEP 2756 H=2756:00 +CPU= 19.462
  16:19:57 STEP 2757 H=2757:00 +CPU= 30.419
  16:20:17 STEP 2758 H=2758:00 +CPU= 19.439
  16:20:36 STEP 2759 H=2759:00 +CPU= 19.360
  16:21:11 STEP 2760 H=2760:00 +CPU= 35.358
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_114.zip
Uploading the intermediate file: upload_file_114.zip
  16:21:30 STEP 2761 H=2761:00 +CPU= 18.892
  16:21:50 STEP 2762 H=2762:00 +CPU= 19.449
  16:22:20 STEP 2763 H=2763:00 +CPU= 30.430
  16:22:40 STEP 2764 H=2764:00 +CPU= 19.461
  16:22:58 STEP 2765 H=2765:00 +CPU= 18.846
  16:23:30 STEP 2766 H=2766:00 +CPU= 31.152
  16:23:48 STEP 2767 H=2767:00 +CPU= 18.866
  16:24:08 STEP 2768 H=2768:00 +CPU= 19.500
  16:24:38 STEP 2769 H=2769:00 +CPU= 30.473
  16:24:58 STEP 2770 H=2770:00 +CPU= 19.498
  16:25:17 STEP 2771 H=2771:00 +CPU= 18.881
  16:25:50 STEP 2772 H=2772:00 +CPU= 33.692
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002772
  16:26:09 STEP 2773 H=2773:00 +CPU= 18.877
  16:26:29 STEP 2774 H=2774:00 +CPU= 19.495
  16:26:59 STEP 2775 H=2775:00 +CPU= 30.432
  16:27:19 STEP 2776 H=2776:00 +CPU= 19.468
  16:27:38 STEP 2777 H=2777:00 +CPU= 18.855
  16:28:09 STEP 2778 H=2778:00 +CPU= 31.165
  16:28:28 STEP 2779 H=2779:00 +CPU= 18.857
  16:28:47 STEP 2780 H=2780:00 +CPU= 19.475
  16:29:18 STEP 2781 H=2781:00 +CPU= 30.457
  16:29:37 STEP 2782 H=2782:00 +CPU= 19.446
  16:29:56 STEP 2783 H=2783:00 +CPU= 19.356
  16:30:32 STEP 2784 H=2784:00 +CPU= 35.313
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_115.zip
Uploading the intermediate file: upload_file_115.zip
  16:30:51 STEP 2785 H=2785:00 +CPU= 18.903
  16:31:10 STEP 2786 H=2786:00 +CPU= 19.468
  16:31:41 STEP 2787 H=2787:00 +CPU= 30.411
  16:32:00 STEP 2788 H=2788:00 +CPU= 19.454
  16:32:19 STEP 2789 H=2789:00 +CPU= 18.878
  16:32:50 STEP 2790 H=2790:00 +CPU= 31.162
  16:33:09 STEP 2791 H=2791:00 +CPU= 18.870
  16:33:28 STEP 2792 H=2792:00 +CPU= 19.499
  16:33:59 STEP 2793 H=2793:00 +CPU= 30.461
  16:34:18 STEP 2794 H=2794:00 +CPU= 19.521
  16:34:37 STEP 2795 H=2795:00 +CPU= 18.905
  16:35:11 STEP 2796 H=2796:00 +CPU= 33.709
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002796
  16:35:30 STEP 2797 H=2797:00 +CPU= 18.910
  16:35:49 STEP 2798 H=2798:00 +CPU= 19.518
  16:36:20 STEP 2799 H=2799:00 +CPU= 30.476
  16:36:39 STEP 2800 H=2800:00 +CPU= 19.505
  16:36:58 STEP 2801 H=2801:00 +CPU= 18.880
  16:37:30 STEP 2802 H=2802:00 +CPU= 31.183
  16:37:48 STEP 2803 H=2803:00 +CPU= 18.874
  16:38:08 STEP 2804 H=2804:00 +CPU= 19.488
  16:38:38 STEP 2805 H=2805:00 +CPU= 30.465
  16:38:58 STEP 2806 H=2806:00 +CPU= 19.472
  16:39:17 STEP 2807 H=2807:00 +CPU= 19.390
  16:39:53 STEP 2808 H=2808:00 +CPU= 35.353
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_116.zip
Uploading the intermediate file: upload_file_116.zip
  16:40:12 STEP 2809 H=2809:00 +CPU= 18.915
  16:40:31 STEP 2810 H=2810:00 +CPU= 19.470
  16:41:02 STEP 2811 H=2811:00 +CPU= 30.473
  16:41:21 STEP 2812 H=2812:00 +CPU= 19.498
  16:41:40 STEP 2813 H=2813:00 +CPU= 18.873
  16:42:11 STEP 2814 H=2814:00 +CPU= 31.188
  16:42:30 STEP 2815 H=2815:00 +CPU= 18.897
  16:42:50 STEP 2816 H=2816:00 +CPU= 19.520
  16:43:20 STEP 2817 H=2817:00 +CPU= 30.489
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0796
(argv4) batchid: 970
(argv5) wuid: 12187440
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_afc53c81a3ca741537c706373f715a24 to: /var/lib/boinc/slots/1/oifs_43r3_ps_0796_2001050100_123_970_12187440.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_ps_0796_2001050100_123_970_12187440.zip
ic_ancil_file: ic_ancil_12187440
ifsdata_file: ifsdata_12187440
climate_data_file: clim_data_12187440
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a89f0741991f6b2c4bc07ced9d3cd0c2 to: /var/lib/boinc/slots/1/ic_ancil_12187440.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12187440.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_e55f366fbb72509f39d366e995cdffa8 to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12187440.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12187440.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12187440.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12187440.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12187440.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12187440.xml
-- Model is restarting --
Adjusting last_iter, 2818, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12187440.xml
last_cpu_time: 67119
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847
The child process has been launched with process id: 6811
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:mondain:1:1:6811:6811] [20221230:172520:1672446320:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  17:25:56 STEP 2808 H=2808:00 +CPU= 33.881
  17:26:14 STEP 2809 H=2809:00 +CPU= 18.883
  17:26:34 STEP 2810 H=2810:00 +CPU= 19.504
  17:27:04 STEP 2811 H=2811:00 +CPU= 30.492
  17:27:24 STEP 2812 H=2812:00 +CPU= 19.505
  17:27:43 STEP 2813 H=2813:00 +CPU= 18.913
  17:28:14 STEP 2814 H=2814:00 +CPU= 31.220
  17:28:33 STEP 2815 H=2815:00 +CPU= 18.920
  17:28:53 STEP 2816 H=2816:00 +CPU= 19.564
  17:29:23 STEP 2817 H=2817:00 +CPU= 30.516
  17:29:43 STEP 2818 H=2818:00 +CPU= 19.550
  17:30:02 STEP 2819 H=2819:00 +CPU= 18.940
  17:30:35 STEP 2820 H=2820:00 +CPU= 33.716
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002820
  17:30:54 STEP 2821 H=2821:00 +CPU= 18.930
  17:31:14 STEP 2822 H=2822:00 +CPU= 19.514
  17:31:44 STEP 2823 H=2823:00 +CPU= 30.462
  17:32:04 STEP 2824 H=2824:00 +CPU= 19.519
  17:32:23 STEP 2825 H=2825:00 +CPU= 18.902
  17:32:54 STEP 2826 H=2826:00 +CPU= 31.209
  17:33:13 STEP 2827 H=2827:00 +CPU= 18.888
  17:33:32 STEP 2828 H=2828:00 +CPU= 19.514
  17:34:03 STEP 2829 H=2829:00 +CPU= 30.489
  17:34:22 STEP 2830 H=2830:00 +CPU= 19.496
  17:34:42 STEP 2831 H=2831:00 +CPU= 19.407
  17:35:17 STEP 2832 H=2832:00 +CPU= 35.279
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12187440/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0796_2001050100_123_970_12187440_0_r1293634847_117.zip
Uploading the intermediate file: upload_file_117.zip
  17:35:36 STEP 2833 H=2833:00 +CPU= 18.927
  17:35:56 STEP 2834 H=2834:00 +CPU= 19.499
  17:36:26 STEP 2835 H=2835:00 +CPU= 30.507
  17:36:46 STEP 2836 H=2836:00 +CPU= 19.510
  17:37:04 STEP 2837 H=2837:00 +CPU= 18.900
  17:37:36 STEP 2838 H=2838:00 +CPU= 31.232
  17:37:55 STEP 2839 H=2839:00 +CPU= 18.918
  17:38:14 STEP 2840 H=2840:00 +CPU= 19.551
  17:38:45 STEP 2841 H=2841:00 +CPU= 30.539
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:03:01 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 7,257,600 47,315 0.0065
09 May 2023 16:02:56 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 6,393,600 41,693 0.0065
09 May 2023 16:02:51 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 5,529,600 36,071 0.0065
09 May 2023 16:02:45 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 4,665,600 30,460 0.0065
09 May 2023 16:02:37 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 3,456,000 22,505 0.0065
09 May 2023 16:02:31 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 2,592,000 16,883 0.0065
09 May 2023 16:02:27 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 1,728,000 11,271 0.0065
09 May 2023 16:02:22 1463165 22272866 oifs_43r3_ps_0796_2001050100_123_970_12187440_0 864,000 5,649 0.0065


©2024 cpdn.org