climateprediction.net home page
Task 22273353

Task 22273353

Name oifs_43r3_ps_0253_2002050100_123_971_12187897_0
Workunit 12187897
Created 23 Dec 2022, 2:15:17 UTC
Sent 18 Jan 2023, 3:22:05 UTC
Report deadline 19 Mar 2023, 3:22:05 UTC
Received 29 Jan 2023, 12:50:47 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1536107
Run time 2 days 3 hours 48 min 18 sec
CPU time 2 days 0 hours 39 min 4 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,410.14 MB
Peak swap size 4,978.16 MB
Peak disk usage 6,062.74 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
2050100_123_971_12187897_0_r2102816723_111.zip
Uploading the intermediate file: upload_file_111.zip
  21:50:21 STEP 2689 H=2689:00 +CPU= 42.392
  21:51:05 STEP 2690 H=2690:00 +CPU= 43.268
  21:52:13 STEP 2691 H=2691:00 +CPU= 67.283
  21:52:57 STEP 2692 H=2692:00 +CPU= 43.661
  21:53:40 STEP 2693 H=2693:00 +CPU= 42.726
  21:54:49 STEP 2694 H=2694:00 +CPU= 69.075
  21:55:32 STEP 2695 H=2695:00 +CPU= 42.717
  21:56:16 STEP 2696 H=2696:00 +CPU= 43.524
  21:57:24 STEP 2697 H=2697:00 +CPU= 67.663
  21:58:08 STEP 2698 H=2698:00 +CPU= 43.654
  21:58:50 STEP 2699 H=2699:00 +CPU= 42.346
  22:00:06 STEP 2700 H=2700:00 +CPU= 74.837
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002700
  22:00:50 STEP 2701 H=2701:00 +CPU= 43.291
  22:01:35 STEP 2702 H=2702:00 +CPU= 44.116
  22:02:45 STEP 2703 H=2703:00 +CPU= 69.486
  22:03:32 STEP 2704 H=2704:00 +CPU= 44.987
  22:04:17 STEP 2705 H=2705:00 +CPU= 43.961
  22:05:30 STEP 2706 H=2706:00 +CPU= 71.469
  22:06:17 STEP 2707 H=2707:00 +CPU= 44.707
  22:07:05 STEP 2708 H=2708:00 +CPU= 45.938
  22:08:19 STEP 2709 H=2709:00 +CPU= 71.861
  22:09:08 STEP 2710 H=2710:00 +CPU= 46.480
  22:09:56 STEP 2711 H=2711:00 +CPU= 46.293
  22:11:24 STEP 2712 H=2712:00 +CPU= 83.604
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_112.zip
Uploading the intermediate file: upload_file_112.zip
  22:12:11 STEP 2713 H=2713:00 +CPU= 45.284
  22:12:57 STEP 2714 H=2714:00 +CPU= 45.474
  22:14:10 STEP 2715 H=2715:00 +CPU= 70.660
  22:14:59 STEP 2716 H=2716:00 +CPU= 46.356
  22:15:46 STEP 2717 H=2717:00 +CPU= 45.169
  22:17:04 STEP 2718 H=2718:00 +CPU= 74.466
  22:17:50 STEP 2719 H=2719:00 +CPU= 44.392
  22:18:39 STEP 2720 H=2720:00 +CPU= 47.269
  22:19:55 STEP 2721 H=2721:00 +CPU= 72.475
  22:20:45 STEP 2722 H=2722:00 +CPU= 47.401
  22:21:34 STEP 2723 H=2723:00 +CPU= 46.353
  22:23:01 STEP 2724 H=2724:00 +CPU= 82.001
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002724
  22:23:48 STEP 2725 H=2725:00 +CPU= 45.543
  22:24:37 STEP 2726 H=2726:00 +CPU= 46.882
  22:25:52 STEP 2727 H=2727:00 +CPU= 72.199
  22:26:42 STEP 2728 H=2728:00 +CPU= 48.000
  22:27:31 STEP 2729 H=2729:00 +CPU= 46.167
  22:28:51 STEP 2730 H=2730:00 +CPU= 75.586
  22:29:36 STEP 2731 H=2731:00 +CPU= 43.464
  22:30:21 STEP 2732 H=2732:00 +CPU= 44.833
  22:31:33 STEP 2733 H=2733:00 +CPU= 70.077
  22:32:21 STEP 2734 H=2734:00 +CPU= 46.423
  22:33:08 STEP 2735 H=2735:00 +CPU= 45.579
  22:34:40 STEP 2736 H=2736:00 +CPU= 82.359
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_113.zip
Uploading the intermediate file: upload_file_113.zip
  22:35:26 STEP 2737 H=2737:00 +CPU= 45.082
  22:36:12 STEP 2738 H=2738:00 +CPU= 45.067
  22:37:25 STEP 2739 H=2739:00 +CPU= 70.858
  22:38:13 STEP 2740 H=2740:00 +CPU= 46.008
  22:38:59 STEP 2741 H=2741:00 +CPU= 44.880
  22:40:11 STEP 2742 H=2742:00 +CPU= 70.708
  22:40:53 STEP 2743 H=2743:00 +CPU= 41.877
  22:41:37 STEP 2744 H=2744:00 +CPU= 43.232
  22:42:44 STEP 2745 H=2745:00 +CPU= 66.964
  22:43:27 STEP 2746 H=2746:00 +CPU= 42.919
  22:44:10 STEP 2747 H=2747:00 +CPU= 42.511
  22:45:26 STEP 2748 H=2748:00 +CPU= 74.931
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002748
  22:46:09 STEP 2749 H=2749:00 +CPU= 42.598
  22:46:53 STEP 2750 H=2750:00 +CPU= 43.386
  22:48:00 STEP 2751 H=2751:00 +CPU= 67.446
  22:48:44 STEP 2752 H=2752:00 +CPU= 43.255
  22:49:26 STEP 2753 H=2753:00 +CPU= 42.010
  22:50:35 STEP 2754 H=2754:00 +CPU= 68.436
  22:51:17 STEP 2755 H=2755:00 +CPU= 41.926
  22:52:01 STEP 2756 H=2756:00 +CPU= 43.555
  22:53:10 STEP 2757 H=2757:00 +CPU= 67.889
  22:53:56 STEP 2758 H=2758:00 +CPU= 45.243
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0253
(argv4) batchid: 971
(argv5) wuid: 12187897
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f2181658f1650845bd008db5673844d5 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0253_2002050100_123_971_12187897.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0253_2002050100_123_971_12187897.zip
ic_ancil_file: ic_ancil_12187897
ifsdata_file: ifsdata_12187897
climate_data_file: clim_data_12187897
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c566bbf2cb05fe105f907602b6e07fcb to: /var/lib/boinc-client/slots/0/ic_ancil_12187897.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12187897.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ab365c71aa764bd1df85fd88e5e809dc to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187897.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187897.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187897.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187897.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12187897.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12187897.xml
-- Model is restarting --
Adjusting last_iter, 2759, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12187897.xml
last_cpu_time: 158885
upload_file_number: 114
last_iter: 2735
last_upload: 9849600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723
The child process has been launched with process id: 169993
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:169993:169993] [20230128:225705:1674971825:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:58:41 STEP 2736 H=2736:00 +CPU= 77.181
  22:59:36 STEP 2737 H=2737:00 +CPU= 52.356
  23:00:40 STEP 2738 H=2738:00 +CPU= 56.740
  23:02:16 STEP 2739 H=2739:00 +CPU= 86.691
  23:03:16 STEP 2740 H=2740:00 +CPU= 55.600
  23:04:16 STEP 2741 H=2741:00 +CPU= 54.292
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:06:32 STEP 2742 H=2742:00 +CPU= 96.332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:08:04 STEP 2743 H=2743:00 +CPU= 61.398
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:09:38 STEP 2744 H=2744:00 +CPU= 63.257
  23:11:48 STEP 2745 H=2745:00 +CPU= 98.875
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:13:33 STEP 2746 H=2746:00 +CPU= 63.216
  23:14:53 STEP 2747 H=2747:00 +CPU= 61.411
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:17:28 STEP 2748 H=2748:00 +CPU=109.198
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002748
  23:18:48 STEP 2749 H=2749:00 +CPU= 61.532
  23:20:10 STEP 2750 H=2750:00 +CPU= 63.051
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:22:28 STEP 2751 H=2751:00 +CPU= 98.161
  23:23:51 STEP 2752 H=2752:00 +CPU= 63.360
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:25:21 STEP 2753 H=2753:00 +CPU= 61.397
  23:27:31 STEP 2754 H=2754:00 +CPU= 99.704
  23:28:50 STEP 2755 H=2755:00 +CPU= 61.230
  23:30:12 STEP 2756 H=2756:00 +CPU= 62.906
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:32:30 STEP 2757 H=2757:00 +CPU= 97.712
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:34:05 STEP 2758 H=2758:00 +CPU= 63.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:35:41 STEP 2759 H=2759:00 +CPU= 63.602
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:38:21 STEP 2760 H=2760:00 +CPU=114.794
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_114.zip
Uploading the intermediate file: upload_file_114.zip
  23:39:41 STEP 2761 H=2761:00 +CPU= 61.428
  23:41:02 STEP 2762 H=2762:00 +CPU= 62.841
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:43:19 STEP 2763 H=2763:00 +CPU= 97.814
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:44:53 STEP 2764 H=2764:00 +CPU= 63.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:46:37 STEP 2765 H=2765:00 +CPU= 61.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:49:04 STEP 2766 H=2766:00 +CPU=103.814
  23:50:25 STEP 2767 H=2767:00 +CPU= 61.656
  23:51:48 STEP 2768 H=2768:00 +CPU= 63.462
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:54:20 STEP 2769 H=2769:00 +CPU= 98.720
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:55:55 STEP 2770 H=2770:00 +CPU= 63.627
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  23:57:27 STEP 2771 H=2771:00 +CPU= 61.578
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:00:01 STEP 2772 H=2772:00 +CPU=109.009
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002772
  00:01:17 STEP 2773 H=2773:00 +CPU= 60.333
  00:02:36 STEP 2774 H=2774:00 +CPU= 62.343
  00:04:43 STEP 2775 H=2775:00 +CPU= 98.060
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:06:12 STEP 2776 H=2776:00 +CPU= 61.629
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:07:44 STEP 2777 H=2777:00 +CPU= 61.644
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:10:19 STEP 2778 H=2778:00 +CPU=100.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:12:16 STEP 2779 H=2779:00 +CPU= 61.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:14:03 STEP 2780 H=2780:00 +CPU= 63.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:16:19 STEP 2781 H=2781:00 +CPU= 97.302
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:17:51 STEP 2782 H=2782:00 +CPU= 62.756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:19:37 STEP 2783 H=2783:00 +CPU= 63.560
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:22:33 STEP 2784 H=2784:00 +CPU=115.361
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_115.zip
Uploading the intermediate file: upload_file_115.zip
  00:23:50 STEP 2785 H=2785:00 +CPU= 60.294
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:25:23 STEP 2786 H=2786:00 +CPU= 62.999
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:27:40 STEP 2787 H=2787:00 +CPU= 97.486
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:29:13 STEP 2788 H=2788:00 +CPU= 63.244
  00:30:34 STEP 2789 H=2789:00 +CPU= 62.309
  00:32:44 STEP 2790 H=2790:00 +CPU=100.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:34:17 STEP 2791 H=2791:00 +CPU= 61.924
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:35:49 STEP 2792 H=2792:00 +CPU= 62.808
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:38:08 STEP 2793 H=2793:00 +CPU= 98.141
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:39:42 STEP 2794 H=2794:00 +CPU= 63.307
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:41:13 STEP 2795 H=2795:00 +CPU= 61.565
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:43:46 STEP 2796 H=2796:00 +CPU=108.794
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002796
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:45:28 STEP 2797 H=2797:00 +CPU= 61.574
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:47:00 STEP 2798 H=2798:00 +CPU= 62.809
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:49:47 STEP 2799 H=2799:00 +CPU= 99.383
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:51:23 STEP 2800 H=2800:00 +CPU= 63.788
  00:52:44 STEP 2801 H=2801:00 +CPU= 61.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:55:08 STEP 2802 H=2802:00 +CPU=100.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:56:41 STEP 2803 H=2803:00 +CPU= 61.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:58:14 STEP 2804 H=2804:00 +CPU= 63.322
  01:00:22 STEP 2805 H=2805:00 +CPU= 98.334
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:01:55 STEP 2806 H=2806:00 +CPU= 63.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:03:30 STEP 2807 H=2807:00 +CPU= 63.728
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:06:11 STEP 2808 H=2808:00 +CPU=114.773
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_116.zip
Uploading the intermediate file: upload_file_116.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:07:33 STEP 2809 H=2809:00 +CPU= 60.217
  01:08:56 STEP 2810 H=2810:00 +CPU= 63.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:11:27 STEP 2811 H=2811:00 +CPU= 97.944
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:13:00 STEP 2812 H=2812:00 +CPU= 62.790
  01:14:21 STEP 2813 H=2813:00 +CPU= 61.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:16:45 STEP 2814 H=2814:00 +CPU=100.083
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0253
(argv4) batchid: 971
(argv5) wuid: 12187897
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f2181658f1650845bd008db5673844d5 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0253_2002050100_123_971_12187897.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0253_2002050100_123_971_12187897.zip
ic_ancil_file: ic_ancil_12187897
ifsdata_file: ifsdata_12187897
climate_data_file: clim_data_12187897
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c566bbf2cb05fe105f907602b6e07fcb to: /var/lib/boinc-client/slots/0/ic_ancil_12187897.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12187897.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ab365c71aa764bd1df85fd88e5e809dc to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187897.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187897.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187897.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187897.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12187897.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12187897.xml
-- Model is restarting --
Adjusting last_iter, 2815, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12187897.xml
last_cpu_time: 164825
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723
The child process has been launched with process id: 179930
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:179930:179930] [20230129:012512:1674980712:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0253
(argv4) batchid: 971
(argv5) wuid: 12187897
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_f2181658f1650845bd008db5673844d5 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0253_2002050100_123_971_12187897.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0253_2002050100_123_971_12187897.zip
ic_ancil_file: ic_ancil_12187897
ifsdata_file: ifsdata_12187897
climate_data_file: clim_data_12187897
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c566bbf2cb05fe105f907602b6e07fcb to: /var/lib/boinc-client/slots/0/ic_ancil_12187897.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12187897.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ab365c71aa764bd1df85fd88e5e809dc to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187897.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12187897.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187897.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12187897.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12187897.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12187897.xml
-- Model is restarting --
Adjusting last_iter, 2808, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12187897.xml
last_cpu_time: 164911
upload_file_number: 117
last_iter: 2807
last_upload: 10108800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723
The child process has been launched with process id: 182160
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Shaynovo:1:1:182160:182160] [20230129:013025:1674981025:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:33:18 STEP 2808 H=2808:00 +CPU=112.990
  01:34:38 STEP 2809 H=2809:00 +CPU= 61.211
  01:35:57 STEP 2810 H=2810:00 +CPU= 62.293
  01:38:01 STEP 2811 H=2811:00 +CPU= 97.589
  01:39:22 STEP 2812 H=2812:00 +CPU= 62.632
  01:40:41 STEP 2813 H=2813:00 +CPU= 61.034
  01:42:50 STEP 2814 H=2814:00 +CPU= 99.735
  01:44:09 STEP 2815 H=2815:00 +CPU= 61.045
  01:45:30 STEP 2816 H=2816:00 +CPU= 62.769
  01:47:36 STEP 2817 H=2817:00 +CPU= 98.020
  01:48:57 STEP 2818 H=2818:00 +CPU= 62.785
  01:50:16 STEP 2819 H=2819:00 +CPU= 61.026
  01:52:34 STEP 2820 H=2820:00 +CPU=108.525
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002820
  01:53:53 STEP 2821 H=2821:00 +CPU= 61.064
  01:55:15 STEP 2822 H=2822:00 +CPU= 63.059
  01:57:21 STEP 2823 H=2823:00 +CPU= 98.007
  01:58:41 STEP 2824 H=2824:00 +CPU= 62.622
  01:59:57 STEP 2825 H=2825:00 +CPU= 60.609
  02:02:02 STEP 2826 H=2826:00 +CPU= 99.060
  02:03:14 STEP 2827 H=2827:00 +CPU= 59.443
  02:04:19 STEP 2828 H=2828:00 +CPU= 58.065
  02:06:00 STEP 2829 H=2829:00 +CPU= 89.975
  02:07:05 STEP 2830 H=2830:00 +CPU= 57.982
  02:08:11 STEP 2831 H=2831:00 +CPU= 58.372
  02:10:12 STEP 2832 H=2832:00 +CPU=105.573
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_117.zip
Uploading the intermediate file: upload_file_117.zip
  02:11:17 STEP 2833 H=2833:00 +CPU= 56.999
  02:12:22 STEP 2834 H=2834:00 +CPU= 57.999
  02:14:03 STEP 2835 H=2835:00 +CPU= 90.602
  02:15:09 STEP 2836 H=2836:00 +CPU= 58.211
  02:16:12 STEP 2837 H=2837:00 +CPU= 56.556
  02:17:56 STEP 2838 H=2838:00 +CPU= 92.207
  02:18:59 STEP 2839 H=2839:00 +CPU= 56.545
  02:20:04 STEP 2840 H=2840:00 +CPU= 58.015
  02:21:46 STEP 2841 H=2841:00 +CPU= 90.435
  02:22:51 STEP 2842 H=2842:00 +CPU= 58.202
  02:23:55 STEP 2843 H=2843:00 +CPU= 56.560
  02:25:47 STEP 2844 H=2844:00 +CPU= 99.956
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002844
  02:26:51 STEP 2845 H=2845:00 +CPU= 56.759
  02:27:56 STEP 2846 H=2846:00 +CPU= 57.860
  02:29:36 STEP 2847 H=2847:00 +CPU= 89.833
  02:30:42 STEP 2848 H=2848:00 +CPU= 58.339
  02:31:45 STEP 2849 H=2849:00 +CPU= 56.023
  02:33:28 STEP 2850 H=2850:00 +CPU= 91.976
  02:34:32 STEP 2851 H=2851:00 +CPU= 56.719
  02:35:37 STEP 2852 H=2852:00 +CPU= 57.891
  02:37:18 STEP 2853 H=2853:00 +CPU= 90.152
  02:38:23 STEP 2854 H=2854:00 +CPU= 57.630
  02:39:28 STEP 2855 H=2855:00 +CPU= 58.311
  02:41:29 STEP 2856 H=2856:00 +CPU=106.049
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_118.zip
Uploading the intermediate file: upload_file_118.zip
  02:42:34 STEP 2857 H=2857:00 +CPU= 57.038
  02:43:39 STEP 2858 H=2858:00 +CPU= 57.889
  02:45:20 STEP 2859 H=2859:00 +CPU= 89.948
  02:46:25 STEP 2860 H=2860:00 +CPU= 58.135
  02:47:28 STEP 2861 H=2861:00 +CPU= 56.331
  02:49:12 STEP 2862 H=2862:00 +CPU= 92.020
  02:50:15 STEP 2863 H=2863:00 +CPU= 56.464
  02:51:20 STEP 2864 H=2864:00 +CPU= 58.037
  02:53:01 STEP 2865 H=2865:00 +CPU= 90.223
  02:54:07 STEP 2866 H=2866:00 +CPU= 58.182
  02:55:10 STEP 2867 H=2867:00 +CPU= 56.341
  02:57:03 STEP 2868 H=2868:00 +CPU=100.276
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002868
  02:58:06 STEP 2869 H=2869:00 +CPU= 56.287
  02:59:10 STEP 2870 H=2870:00 +CPU= 57.949
  03:00:52 STEP 2871 H=2871:00 +CPU= 90.508
  03:01:58 STEP 2872 H=2872:00 +CPU= 58.141
  03:03:01 STEP 2873 H=2873:00 +CPU= 56.576
  03:04:45 STEP 2874 H=2874:00 +CPU= 92.135
  03:05:48 STEP 2875 H=2875:00 +CPU= 56.065
  03:06:53 STEP 2876 H=2876:00 +CPU= 58.187
  03:08:34 STEP 2877 H=2877:00 +CPU= 90.073
  03:09:40 STEP 2878 H=2878:00 +CPU= 58.110
  03:10:45 STEP 2879 H=2879:00 +CPU= 57.956
  03:12:46 STEP 2880 H=2880:00 +CPU=105.890
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_119.zip
Uploading the intermediate file: upload_file_119.zip
  03:13:50 STEP 2881 H=2881:00 +CPU= 56.526
  03:14:56 STEP 2882 H=2882:00 +CPU= 58.083
  03:16:37 STEP 2883 H=2883:00 +CPU= 90.152
  03:17:42 STEP 2884 H=2884:00 +CPU= 57.961
  03:18:45 STEP 2885 H=2885:00 +CPU= 56.519
  03:20:29 STEP 2886 H=2886:00 +CPU= 91.923
  03:21:32 STEP 2887 H=2887:00 +CPU= 56.657
  03:22:37 STEP 2888 H=2888:00 +CPU= 57.910
  03:24:18 STEP 2889 H=2889:00 +CPU= 89.888
  03:25:23 STEP 2890 H=2890:00 +CPU= 58.125
  03:26:26 STEP 2891 H=2891:00 +CPU= 56.375
  03:28:19 STEP 2892 H=2892:00 +CPU=100.267
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002892
  03:29:23 STEP 2893 H=2893:00 +CPU= 56.560
  03:30:28 STEP 2894 H=2894:00 +CPU= 58.174
  03:32:10 STEP 2895 H=2895:00 +CPU= 90.337
  03:33:14 STEP 2896 H=2896:00 +CPU= 57.809
  03:34:18 STEP 2897 H=2897:00 +CPU= 56.226
  03:36:01 STEP 2898 H=2898:00 +CPU= 91.683
  03:37:04 STEP 2899 H=2899:00 +CPU= 56.529
  03:38:09 STEP 2900 H=2900:00 +CPU= 57.905
  03:39:49 STEP 2901 H=2901:00 +CPU= 89.803
  03:40:54 STEP 2902 H=2902:00 +CPU= 57.912
  03:42:00 STEP 2903 H=2903:00 +CPU= 58.412
  03:44:00 STEP 2904 H=2904:00 +CPU=105.256
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_120.zip
Uploading the intermediate file: upload_file_120.zip
  03:45:05 STEP 2905 H=2905:00 +CPU= 56.916
  03:46:10 STEP 2906 H=2906:00 +CPU= 58.063
  03:47:52 STEP 2907 H=2907:00 +CPU= 90.612
  03:48:58 STEP 2908 H=2908:00 +CPU= 58.214
  03:50:01 STEP 2909 H=2909:00 +CPU= 56.359
  03:51:44 STEP 2910 H=2910:00 +CPU= 92.093
  03:52:48 STEP 2911 H=2911:00 +CPU= 56.385
  03:53:53 STEP 2912 H=2912:00 +CPU= 57.949
  03:55:33 STEP 2913 H=2913:00 +CPU= 90.150
  03:56:39 STEP 2914 H=2914:00 +CPU= 58.263
  03:57:42 STEP 2915 H=2915:00 +CPU= 56.455
  03:59:34 STEP 2916 H=2916:00 +CPU= 99.896
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002916
  04:00:38 STEP 2917 H=2917:00 +CPU= 56.756
  04:01:43 STEP 2918 H=2918:00 +CPU= 58.115
  04:03:25 STEP 2919 H=2919:00 +CPU= 90.426
  04:04:31 STEP 2920 H=2920:00 +CPU= 58.038
  04:05:34 STEP 2921 H=2921:00 +CPU= 56.417
  04:07:18 STEP 2922 H=2922:00 +CPU= 92.182
  04:08:22 STEP 2923 H=2923:00 +CPU= 56.688
  04:09:27 STEP 2924 H=2924:00 +CPU= 57.879
  04:11:08 STEP 2925 H=2925:00 +CPU= 89.991
  04:12:13 STEP 2926 H=2926:00 +CPU= 57.989
  04:13:19 STEP 2927 H=2927:00 +CPU= 58.356
  04:15:19 STEP 2928 H=2928:00 +CPU=105.966
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_121.zip
Uploading the intermediate file: upload_file_121.zip
  04:16:23 STEP 2929 H=2929:00 +CPU= 56.902
  04:17:28 STEP 2930 H=2930:00 +CPU= 57.859
  04:19:09 STEP 2931 H=2931:00 +CPU= 90.157
  04:20:15 STEP 2932 H=2932:00 +CPU= 57.987
  04:21:18 STEP 2933 H=2933:00 +CPU= 56.458
  04:23:02 STEP 2934 H=2934:00 +CPU= 92.191
  04:24:06 STEP 2935 H=2935:00 +CPU= 56.579
  04:25:10 STEP 2936 H=2936:00 +CPU= 57.974
  04:26:51 STEP 2937 H=2937:00 +CPU= 90.281
  04:27:56 STEP 2938 H=2938:00 +CPU= 58.129
  04:29:00 STEP 2939 H=2939:00 +CPU= 56.340
  04:30:53 STEP 2940 H=2940:00 +CPU=100.316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002940
  04:31:56 STEP 2941 H=2941:00 +CPU= 56.390
  04:33:01 STEP 2942 H=2942:00 +CPU= 58.214
  04:34:43 STEP 2943 H=2943:00 +CPU= 90.207
  04:35:47 STEP 2944 H=2944:00 +CPU= 57.692
  04:36:51 STEP 2945 H=2945:00 +CPU= 56.519
  04:38:34 STEP 2946 H=2946:00 +CPU= 91.818
  04:39:37 STEP 2947 H=2947:00 +CPU= 56.516
  04:40:42 STEP 2948 H=2948:00 +CPU= 57.839
  04:42:24 STEP 2949 H=2949:00 +CPU= 90.381
  04:43:29 STEP 2950 H=2950:00 +CPU= 57.719
  04:44:34 STEP 2951 H=2951:00 +CPU= 58.415
  04:46:32 STEP 2952 H=2952:00 +CPU=102.417
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187897/ICMSHhq0f+002952
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0253_2002050100_123_971_12187897_0_r2102816723_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
04:49:58 (182101): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:47:44 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 10,623,600 171,769 0.0162
09 May 2023 16:44:55 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 10,623,600 175,144 0.0165
09 May 2023 16:44:42 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 9,158,400 147,375 0.0161
09 May 2023 16:44:26 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 7,516,800 118,542 0.0158
09 May 2023 16:43:42 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 6,048,000 96,447 0.0159
09 May 2023 16:43:33 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 5,184,000 83,797 0.0162
09 May 2023 16:43:24 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 4,320,000 70,709 0.0164
09 May 2023 16:43:13 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 3,456,000 57,642 0.0167
09 May 2023 16:43:03 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 2,592,000 44,276 0.0171
09 May 2023 16:42:48 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 1,728,000 30,600 0.0177
09 May 2023 16:42:37 1536107 22273353 oifs_43r3_ps_0253_2002050100_123_971_12187897_0 864,000 17,204 0.0199


©2024 cpdn.org