climateprediction.net home page
Task 22264399

Task 22264399

Name oifs_43r3_ps_0843_1993050100_123_962_12179487_0
Workunit 12179487
Created 22 Dec 2022, 4:10:49 UTC
Sent 22 Dec 2022, 4:33:06 UTC
Report deadline 21 Jan 2023, 4:33:06 UTC
Received 29 Dec 2022, 22:34:51 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1527792
Run time 16 hours 7 min 41 sec
CPU time 9 hours 57 min 57 sec
Validate state Invalid
Credit 7,373.52
Device peak FLOPS 6.83 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.14 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,643.00 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
t/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3993 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                CLOUD_LAYER 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005047:1672185047:185.658] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 CLOUDSC 
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005048:1672185048:186.658] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2877:2877] [20221228:005048:1672185048:186.658] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000101A2B9  cloudsc_                  774  cloudsc.F90
oifs_43r3_model.e  0000000001014D57  cloud_layer_              131  cloud_layer.F90
oifs_43r3_model.e  0000000000F1A790  callpar_                 1223  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0843
(argv4) batchid: 962
(argv5) wuid: 12179487
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8c72e395c84d8b696f8cbf6c940e9d29 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0843_1993050100_123_962_12179487.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0843_1993050100_123_962_12179487.zip
ic_ancil_file: ic_ancil_12179487
ifsdata_file: ifsdata_12179487
climate_data_file: clim_data_12179487
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c6d64edd46c61c4c21f456a6325a1196 to: /var/lib/boinc-client/slots/6/ic_ancil_12179487.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12179487.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9883cbe597c1ef7f39205eb0f3321282 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179487.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179487.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179487.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179487.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12179487.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12179487.xml
-- Model is restarting --
Adjusting last_iter, 1228, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12179487.xml
last_cpu_time: 27048
upload_file_number: 43
last_iter: 1223
last_upload: 4402800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728
The child process has been launched with process id: 2851
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:005223:1672185143:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:53:23 STEP 1224 H=1224:00 +CPU= 37.695
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:54:12 STEP 1225 H=1225:00 +CPU= 22.899
  00:54:39 STEP 1226 H=1226:00 +CPU= 24.478
  00:55:19 STEP 1227 H=1227:00 +CPU= 37.105
  00:55:44 STEP 1228 H=1228:00 +CPU= 22.610
  00:56:07 STEP 1229 H=1229:00 +CPU= 21.563
  00:56:42 STEP 1230 H=1230:00 +CPU= 33.880
  00:57:06 STEP 1231 H=1231:00 +CPU= 22.279
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  00:57:53 STEP 1232 H=1232:00 +CPU= 22.606
  00:58:29 STEP 1233 H=1233:00 +CPU= 33.302
  00:58:54 STEP 1234 H=1234:00 +CPU= 22.346
  00:59:18 STEP 1235 H=1235:00 +CPU= 22.021
  00:59:59 STEP 1236 H=1236:00 +CPU= 37.353
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001236
  01:00:24 STEP 1237 H=1237:00 +CPU= 22.052
  01:00:49 STEP 1238 H=1238:00 +CPU= 22.538
  01:01:25 STEP 1239 H=1239:00 +CPU= 33.206
  01:01:50 STEP 1240 H=1240:00 +CPU= 22.218
  01:02:14 STEP 1241 H=1241:00 +CPU= 21.898
  01:02:51 STEP 1242 H=1242:00 +CPU= 34.038
  01:03:16 STEP 1243 H=1243:00 +CPU= 21.982
  01:03:40 STEP 1244 H=1244:00 +CPU= 22.692
  01:04:18 STEP 1245 H=1245:00 +CPU= 33.950
  01:04:42 STEP 1246 H=1246:00 +CPU= 21.974
  01:05:07 STEP 1247 H=1247:00 +CPU= 22.485
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_43.zip
Uploading the intermediate file: upload_file_43.zip
  01:05:51 STEP 1248 H=1248:00 +CPU= 39.257
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001248
  01:06:15 STEP 1249 H=1249:00 +CPU= 21.853
  01:06:40 STEP 1250 H=1250:00 +CPU= 22.800
  01:07:17 STEP 1251 H=1251:00 +CPU= 33.805
  01:07:42 STEP 1252 H=1252:00 +CPU= 22.725
  01:08:07 STEP 1253 H=1253:00 +CPU= 22.415
  01:08:45 STEP 1254 H=1254:00 +CPU= 34.317
  01:09:09 STEP 1255 H=1255:00 +CPU= 21.791
  01:09:33 STEP 1256 H=1256:00 +CPU= 22.037
  01:10:07 STEP 1257 H=1257:00 +CPU= 32.144
  01:10:31 STEP 1258 H=1258:00 +CPU= 22.118
  01:10:54 STEP 1259 H=1259:00 +CPU= 21.345
  01:11:32 STEP 1260 H=1260:00 +CPU= 36.303
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001260
  01:11:55 STEP 1261 H=1261:00 +CPU= 21.382
  01:12:19 STEP 1262 H=1262:00 +CPU= 22.137
  01:12:53 STEP 1263 H=1263:00 +CPU= 32.271
  01:13:16 STEP 1264 H=1264:00 +CPU= 21.970
  01:13:41 STEP 1265 H=1265:00 +CPU= 21.858
  01:14:19 STEP 1266 H=1266:00 +CPU= 33.988
  01:14:43 STEP 1267 H=1267:00 +CPU= 22.397
  01:15:08 STEP 1268 H=1268:00 +CPU= 22.905
  01:15:45 STEP 1269 H=1269:00 +CPU= 33.470
  01:16:10 STEP 1270 H=1270:00 +CPU= 22.785
  01:16:36 STEP 1271 H=1271:00 +CPU= 22.684
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_44.zip
Uploading the intermediate file: upload_file_44.zip
  01:17:18 STEP 1272 H=1272:00 +CPU= 38.223
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001272
  01:17:42 STEP 1273 H=1273:00 +CPU= 21.751
  01:18:08 STEP 1274 H=1274:00 +CPU= 22.705
  01:18:45 STEP 1275 H=1275:00 +CPU= 33.912
  01:19:11 STEP 1276 H=1276:00 +CPU= 23.197
  01:19:35 STEP 1277 H=1277:00 +CPU= 22.025
  01:20:14 STEP 1278 H=1278:00 +CPU= 34.494
  01:20:38 STEP 1279 H=1279:00 +CPU= 22.392
  01:21:04 STEP 1280 H=1280:00 +CPU= 22.841
  01:21:41 STEP 1281 H=1281:00 +CPU= 33.937
  01:22:06 STEP 1282 H=1282:00 +CPU= 22.406
  01:22:31 STEP 1283 H=1283:00 +CPU= 22.121
  01:23:13 STEP 1284 H=1284:00 +CPU= 37.386
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001284
  01:23:38 STEP 1285 H=1285:00 +CPU= 22.660
  01:24:04 STEP 1286 H=1286:00 +CPU= 23.285
  01:24:41 STEP 1287 H=1287:00 +CPU= 33.908
  01:25:07 STEP 1288 H=1288:00 +CPU= 23.073
  01:25:32 STEP 1289 H=1289:00 +CPU= 22.310
  01:26:10 STEP 1290 H=1290:00 +CPU= 34.269
  01:26:35 STEP 1291 H=1291:00 +CPU= 22.779
  01:27:01 STEP 1292 H=1292:00 +CPU= 23.160
  01:27:38 STEP 1293 H=1293:00 +CPU= 33.536
  01:28:03 STEP 1294 H=1294:00 +CPU= 22.920
  01:28:29 STEP 1295 H=1295:00 +CPU= 23.117
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_45.zip
Uploading the intermediate file: upload_file_45.zip
  01:29:13 STEP 1296 H=1296:00 +CPU= 39.944
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001296
  01:29:38 STEP 1297 H=1297:00 +CPU= 22.719
  01:30:04 STEP 1298 H=1298:00 +CPU= 23.133
  01:30:42 STEP 1299 H=1299:00 +CPU= 34.503
  01:31:08 STEP 1300 H=1300:00 +CPU= 23.252
  01:31:34 STEP 1301 H=1301:00 +CPU= 23.274
  01:32:12 STEP 1302 H=1302:00 +CPU= 34.260
  01:32:36 STEP 1303 H=1303:00 +CPU= 22.468
  01:33:02 STEP 1304 H=1304:00 +CPU= 22.887
  01:33:39 STEP 1305 H=1305:00 +CPU= 33.621
  01:34:05 STEP 1306 H=1306:00 +CPU= 22.987
  01:34:29 STEP 1307 H=1307:00 +CPU= 22.308
  01:35:08 STEP 1308 H=1308:00 +CPU= 36.211
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001308
  01:35:34 STEP 1309 H=1309:00 +CPU= 22.808
  01:36:00 STEP 1310 H=1310:00 +CPU= 23.436
  01:36:38 STEP 1311 H=1311:00 +CPU= 34.493
  01:37:04 STEP 1312 H=1312:00 +CPU= 23.554
  01:37:30 STEP 1313 H=1313:00 +CPU= 22.907
  01:38:06 STEP 1314 H=1314:00 +CPU= 33.934
  01:38:31 STEP 1315 H=1315:00 +CPU= 22.317
  01:38:57 STEP 1316 H=1316:00 +CPU= 23.370
  01:39:35 STEP 1317 H=1317:00 +CPU= 33.885
  01:40:01 STEP 1318 H=1318:00 +CPU= 23.050
  01:40:27 STEP 1319 H=1319:00 +CPU= 23.265
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_46.zip
Uploading the intermediate file: upload_file_46.zip
  01:41:10 STEP 1320 H=1320:00 +CPU= 39.377
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001320
  01:41:35 STEP 1321 H=1321:00 +CPU= 22.082
  01:42:01 STEP 1322 H=1322:00 +CPU= 23.283
  01:42:39 STEP 1323 H=1323:00 +CPU= 34.232
  01:43:03 STEP 1324 H=1324:00 +CPU= 22.706
  01:43:29 STEP 1325 H=1325:00 +CPU= 22.821
  01:44:08 STEP 1326 H=1326:00 +CPU= 35.258
  01:44:33 STEP 1327 H=1327:00 +CPU= 22.443
  01:44:59 STEP 1328 H=1328:00 +CPU= 23.275
  01:45:37 STEP 1329 H=1329:00 +CPU= 33.833
  01:46:02 STEP 1330 H=1330:00 +CPU= 22.848
  01:46:28 STEP 1331 H=1331:00 +CPU= 23.050
  01:47:10 STEP 1332 H=1332:00 +CPU= 38.239
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001332
  01:47:35 STEP 1333 H=1333:00 +CPU= 22.590
  01:47:59 STEP 1334 H=1334:00 +CPU= 22.774
  01:48:36 STEP 1335 H=1335:00 +CPU= 33.664
  01:49:03 STEP 1336 H=1336:00 +CPU= 23.553
  01:49:28 STEP 1337 H=1337:00 +CPU= 23.036
  01:50:07 STEP 1338 H=1338:00 +CPU= 34.323
  01:50:32 STEP 1339 H=1339:00 +CPU= 22.167
  01:50:58 STEP 1340 H=1340:00 +CPU= 23.306
  01:51:36 STEP 1341 H=1341:00 +CPU= 34.754
  01:52:02 STEP 1342 H=1342:00 +CPU= 23.382
  01:52:28 STEP 1343 H=1343:00 +CPU= 23.682
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_47.zip
Uploading the intermediate file: upload_file_47.zip
  01:53:06 STEP 1344 H=1344:00 +CPU= 33.945
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001344
  01:53:31 STEP 1345 H=1345:00 +CPU= 21.827
  01:53:56 STEP 1346 H=1346:00 +CPU= 23.198
  01:54:36 STEP 1347 H=1347:00 +CPU= 35.416
  01:55:02 STEP 1348 H=1348:00 +CPU= 22.959
  01:55:28 STEP 1349 H=1349:00 +CPU= 23.122
  01:56:08 STEP 1350 H=1350:00 +CPU= 36.189
  01:56:32 STEP 1351 H=1351:00 +CPU= 21.856
  01:56:57 STEP 1352 H=1352:00 +CPU= 23.420
  01:57:36 STEP 1353 H=1353:00 +CPU= 34.882
  01:58:01 STEP 1354 H=1354:00 +CPU= 22.269
  01:58:26 STEP 1355 H=1355:00 +CPU= 22.708
  01:59:09 STEP 1356 H=1356:00 +CPU= 38.312
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001356
  01:59:34 STEP 1357 H=1357:00 +CPU= 22.511
  02:00:02 STEP 1358 H=1358:00 +CPU= 24.879
  02:00:43 STEP 1359 H=1359:00 +CPU= 37.111
  02:01:09 STEP 1360 H=1360:00 +CPU= 24.322
  02:01:36 STEP 1361 H=1361:00 +CPU= 24.565
  02:02:18 STEP 1362 H=1362:00 +CPU= 37.923
  02:02:45 STEP 1363 H=1363:00 +CPU= 24.504
  02:03:14 STEP 1364 H=1364:00 +CPU= 25.881
  02:03:55 STEP 1365 H=1365:00 +CPU= 37.155
  02:04:22 STEP 1366 H=1366:00 +CPU= 23.984
  02:04:49 STEP 1367 H=1367:00 +CPU= 24.625
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_48.zip
Uploading the intermediate file: upload_file_48.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:05:48 STEP 1368 H=1368:00 +CPU= 42.996
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001368
  02:06:14 STEP 1369 H=1369:00 +CPU= 23.702
  02:06:42 STEP 1370 H=1370:00 +CPU= 25.037
  02:07:22 STEP 1371 H=1371:00 +CPU= 36.011
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:08:03 STEP 1372 H=1372:00 +CPU= 26.423
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:08:43 STEP 1373 H=1373:00 +CPU= 25.878
  02:09:27 STEP 1374 H=1374:00 +CPU= 39.264
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:10:17 STEP 1375 H=1375:00 +CPU= 24.340
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:10:57 STEP 1376 H=1376:00 +CPU= 26.030
  02:11:38 STEP 1377 H=1377:00 +CPU= 37.561
  02:12:07 STEP 1378 H=1378:00 +CPU= 26.037
  02:12:36 STEP 1379 H=1379:00 +CPU= 25.824
  02:13:22 STEP 1380 H=1380:00 +CPU= 41.548
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001380
  02:13:49 STEP 1381 H=1381:00 +CPU= 24.580
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:14:31 STEP 1382 H=1382:00 +CPU= 26.867
  02:15:13 STEP 1383 H=1383:00 +CPU= 37.851
  02:15:42 STEP 1384 H=1384:00 +CPU= 25.783
  02:16:09 STEP 1385 H=1385:00 +CPU= 25.033
  02:16:51 STEP 1386 H=1386:00 +CPU= 37.906
  02:17:19 STEP 1387 H=1387:00 +CPU= 25.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:17:58 STEP 1388 H=1388:00 +CPU= 25.121
  02:18:40 STEP 1389 H=1389:00 +CPU= 37.770
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:19:19 STEP 1390 H=1390:00 +CPU= 25.143
  02:19:48 STEP 1391 H=1391:00 +CPU= 25.619
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_49.zip
Uploading the intermediate file: upload_file_49.zip
  02:20:35 STEP 1392 H=1392:00 +CPU= 42.879
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001392
  02:21:03 STEP 1393 H=1393:00 +CPU= 25.050
  02:21:32 STEP 1394 H=1394:00 +CPU= 26.289
  02:22:14 STEP 1395 H=1395:00 +CPU= 37.598
  02:22:41 STEP 1396 H=1396:00 +CPU= 24.786
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:23:21 STEP 1397 H=1397:00 +CPU= 25.273
  02:24:03 STEP 1398 H=1398:00 +CPU= 38.341
  02:24:30 STEP 1399 H=1399:00 +CPU= 24.176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:25:10 STEP 1400 H=1400:00 +CPU= 25.925
  02:25:52 STEP 1401 H=1401:00 +CPU= 37.494
  02:26:20 STEP 1402 H=1402:00 +CPU= 25.040
  02:26:47 STEP 1403 H=1403:00 +CPU= 24.800
  02:27:32 STEP 1404 H=1404:00 +CPU= 40.806
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001404
  02:27:59 STEP 1405 H=1405:00 +CPU= 24.886
  02:28:28 STEP 1406 H=1406:00 +CPU= 25.850
  02:29:08 STEP 1407 H=1407:00 +CPU= 36.532
  02:29:38 STEP 1408 H=1408:00 +CPU= 26.463
  02:30:07 STEP 1409 H=1409:00 +CPU= 25.365
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:31:10 STEP 1410 H=1410:00 +CPU= 37.126
  02:31:37 STEP 1411 H=1411:00 +CPU= 24.927
  02:32:05 STEP 1412 H=1412:00 +CPU= 25.367
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:32:56 STEP 1413 H=1413:00 +CPU= 36.621
  02:33:23 STEP 1414 H=1414:00 +CPU= 25.353
  02:33:52 STEP 1415 H=1415:00 +CPU= 26.008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_50.zip
Uploading the intermediate file: upload_file_50.zip
  02:34:41 STEP 1416 H=1416:00 +CPU= 43.134
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001416
  02:35:09 STEP 1417 H=1417:00 +CPU= 25.098
  02:35:38 STEP 1418 H=1418:00 +CPU= 25.328
  02:36:17 STEP 1419 H=1419:00 +CPU= 35.606
  02:36:45 STEP 1420 H=1420:00 +CPU= 25.353
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:37:25 STEP 1421 H=1421:00 +CPU= 25.908
  02:38:06 STEP 1422 H=1422:00 +CPU= 37.572
  02:38:33 STEP 1423 H=1423:00 +CPU= 24.399
  02:39:03 STEP 1424 H=1424:00 +CPU= 26.310
  02:39:44 STEP 1425 H=1425:00 +CPU= 36.926
  02:40:14 STEP 1426 H=1426:00 +CPU= 26.112
  02:40:43 STEP 1427 H=1427:00 +CPU= 25.695
  02:41:28 STEP 1428 H=1428:00 +CPU= 40.903
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001428
  02:41:56 STEP 1429 H=1429:00 +CPU= 25.165
  02:42:24 STEP 1430 H=1430:00 +CPU= 25.438
  02:43:03 STEP 1431 H=1431:00 +CPU= 34.993
  02:43:31 STEP 1432 H=1432:00 +CPU= 25.141
  02:44:00 STEP 1433 H=1433:00 +CPU= 25.654
  02:44:41 STEP 1434 H=1434:00 +CPU= 37.278
  02:45:06 STEP 1435 H=1435:00 +CPU= 22.999
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:45:45 STEP 1436 H=1436:00 +CPU= 24.708
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:46:36 STEP 1437 H=1437:00 +CPU= 36.634
  02:47:04 STEP 1438 H=1438:00 +CPU= 25.174
  02:47:33 STEP 1439 H=1439:00 +CPU= 26.230
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_51.zip
Uploading the intermediate file: upload_file_51.zip
  02:48:18 STEP 1440 H=1440:00 +CPU= 42.117
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001440
  02:48:46 STEP 1441 H=1441:00 +CPU= 25.843
  02:49:15 STEP 1442 H=1442:00 +CPU= 26.618
  02:49:59 STEP 1443 H=1443:00 +CPU= 39.075
  02:50:28 STEP 1444 H=1444:00 +CPU= 27.105
  02:50:58 STEP 1445 H=1445:00 +CPU= 26.964
  02:51:42 STEP 1446 H=1446:00 +CPU= 39.833
  02:52:11 STEP 1447 H=1447:00 +CPU= 27.058
  02:52:42 STEP 1448 H=1448:00 +CPU= 27.724
  02:53:26 STEP 1449 H=1449:00 +CPU= 39.895
  02:53:58 STEP 1450 H=1450:00 +CPU= 27.714
  02:54:29 STEP 1451 H=1451:00 +CPU= 26.886
  02:55:18 STEP 1452 H=1452:00 +CPU= 43.888
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001452
  02:55:47 STEP 1453 H=1453:00 +CPU= 26.573
  02:56:18 STEP 1454 H=1454:00 +CPU= 28.226
  02:57:02 STEP 1455 H=1455:00 +CPU= 39.877
  02:57:32 STEP 1456 H=1456:00 +CPU= 27.613
  02:58:03 STEP 1457 H=1457:00 +CPU= 27.371
  02:58:46 STEP 1458 H=1458:00 +CPU= 40.921
  02:59:15 STEP 1459 H=1459:00 +CPU= 26.728
  02:59:47 STEP 1460 H=1460:00 +CPU= 28.110
  03:00:32 STEP 1461 H=1461:00 +CPU= 40.344
  03:01:02 STEP 1462 H=1462:00 +CPU= 27.605
  03:01:33 STEP 1463 H=1463:00 +CPU= 28.214
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_52.zip
Uploading the intermediate file: upload_file_52.zip
Uploading trickle at timestep: 5266800
  03:02:22 STEP 1464 H=1464:00 +CPU= 45.486
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001464
  03:02:52 STEP 1465 H=1465:00 +CPU= 26.687
  03:03:23 STEP 1466 H=1466:00 +CPU= 27.985
  03:04:06 STEP 1467 H=1467:00 +CPU= 39.579
  03:04:37 STEP 1468 H=1468:00 +CPU= 27.414
  03:05:07 STEP 1469 H=1469:00 +CPU= 26.726
  03:05:53 STEP 1470 H=1470:00 +CPU= 41.348
  03:06:22 STEP 1471 H=1471:00 +CPU= 26.556
  03:06:53 STEP 1472 H=1472:00 +CPU= 28.461
  03:07:38 STEP 1473 H=1473:00 +CPU= 40.920
  03:08:10 STEP 1474 H=1474:00 +CPU= 28.641
  03:08:41 STEP 1475 H=1475:00 +CPU= 27.500
  03:09:32 STEP 1476 H=1476:00 +CPU= 45.361
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001476
  03:10:02 STEP 1477 H=1477:00 +CPU= 27.080
  03:10:31 STEP 1478 H=1478:00 +CPU= 27.321
  03:11:15 STEP 1479 H=1479:00 +CPU= 39.077
  03:11:46 STEP 1480 H=1480:00 +CPU= 27.464
  03:12:16 STEP 1481 H=1481:00 +CPU= 27.202
  03:13:02 STEP 1482 H=1482:00 +CPU= 40.998
  03:13:32 STEP 1483 H=1483:00 +CPU= 27.089
  03:14:03 STEP 1484 H=1484:00 +CPU= 27.409
  03:14:47 STEP 1485 H=1485:00 +CPU= 39.898
  03:15:18 STEP 1486 H=1486:00 +CPU= 27.628
  03:15:49 STEP 1487 H=1487:00 +CPU= 28.166
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_53.zip
Uploading the intermediate file: upload_file_53.zip
  03:16:40 STEP 1488 H=1488:00 +CPU= 46.201
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001488
  03:17:09 STEP 1489 H=1489:00 +CPU= 26.766
  03:17:39 STEP 1490 H=1490:00 +CPU= 27.810
  03:18:23 STEP 1491 H=1491:00 +CPU= 40.451
  03:18:55 STEP 1492 H=1492:00 +CPU= 28.446
  03:19:27 STEP 1493 H=1493:00 +CPU= 27.548
  03:20:14 STEP 1494 H=1494:00 +CPU= 40.938
  03:20:44 STEP 1495 H=1495:00 +CPU= 27.121
  03:21:15 STEP 1496 H=1496:00 +CPU= 28.151
  03:21:59 STEP 1497 H=1497:00 +CPU= 39.969
  03:22:29 STEP 1498 H=1498:00 +CPU= 27.783
  03:22:59 STEP 1499 H=1499:00 +CPU= 27.246
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.640] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3996MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3996 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS CLDPP T/S    (1002) 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.656] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.657] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                STOCHPERT_LAYER 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032339:1672194219:9076.657] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 SPPTEN 
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032340:1672194220:9077.657] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2851:2851] [20221228:032340:1672194220:9077.657] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000010C3C24  sppten_                   384  sppten.F90
oifs_43r3_model.e  00000000010BD01C  stochpert_layer_          159  stochpert_layer.F90
oifs_43r3_model.e  0000000000F251B4  callpar_                 1554  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1993050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0843
(argv4) batchid: 962
(argv5) wuid: 12179487
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_8c72e395c84d8b696f8cbf6c940e9d29 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0843_1993050100_123_962_12179487.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0843_1993050100_123_962_12179487.zip
ic_ancil_file: ic_ancil_12179487
ifsdata_file: ifsdata_12179487
climate_data_file: clim_data_12179487
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c6d64edd46c61c4c21f456a6325a1196 to: /var/lib/boinc-client/slots/6/ic_ancil_12179487.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12179487.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9883cbe597c1ef7f39205eb0f3321282 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179487.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12179487.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179487.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12179487.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12179487.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12179487.xml
-- Model is restarting --
Adjusting last_iter, 1500, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12179487.xml
last_cpu_time: 35022
upload_file_number: 54
last_iter: 1487
last_upload: 5353200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728
The child process has been launched with process id: 2937
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2937:2937] [20221229:221449:1672348489:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:15:40 STEP 1488 H=1488:00 +CPU= 39.587
  22:16:02 STEP 1489 H=1489:00 +CPU= 21.954
  22:16:26 STEP 1490 H=1490:00 +CPU= 22.797
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:17:50 STEP 1491 H=1491:00 +CPU= 39.021
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:18:32 STEP 1492 H=1492:00 +CPU= 26.942
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:19:13 STEP 1493 H=1493:00 +CPU= 26.316
  22:19:57 STEP 1494 H=1494:00 +CPU= 40.014
  22:20:27 STEP 1495 H=1495:00 +CPU= 26.364
  22:21:01 STEP 1496 H=1496:00 +CPU= 26.961
  22:21:44 STEP 1497 H=1497:00 +CPU= 38.862
  22:22:14 STEP 1498 H=1498:00 +CPU= 27.164
  22:22:42 STEP 1499 H=1499:00 +CPU= 26.130
  22:23:29 STEP 1500 H=1500:00 +CPU= 43.078
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001500
  22:23:59 STEP 1501 H=1501:00 +CPU= 26.900
  22:24:28 STEP 1502 H=1502:00 +CPU= 26.874
  22:25:15 STEP 1503 H=1503:00 +CPU= 38.246
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:26:01 STEP 1504 H=1504:00 +CPU= 26.331
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:27:34 STEP 1505 H=1505:00 +CPU= 26.133
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:28:34 STEP 1506 H=1506:00 +CPU= 40.495
  22:29:04 STEP 1507 H=1507:00 +CPU= 26.368
  22:29:35 STEP 1508 H=1508:00 +CPU= 27.037
  22:30:45 STEP 1509 H=1509:00 +CPU= 40.171
  22:31:35 STEP 1510 H=1510:00 +CPU= 27.816
  22:32:08 STEP 1511 H=1511:00 +CPU= 27.600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12179487/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0843_1993050100_123_962_12179487_0_r998197728_54.zip
Uploading the intermediate file: upload_file_54.zip
  22:32:55 STEP 1512 H=1512:00 +CPU= 41.436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001512
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  22:33:52 STEP 1513 H=1513:00 +CPU= 27.505
  22:34:28 STEP 1514 H=1514:00 +CPU= 29.147
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:59:28 1527792 22264399 oifs_43r3_ps_0843_1993050100_123_962_12179487_0 5,266,800 33,835 0.0064
09 May 2023 15:59:18 1527792 22264399 oifs_43r3_ps_0843_1993050100_123_962_12179487_0 4,230,000 24,885 0.0059
09 May 2023 15:59:10 1527792 22264399 oifs_43r3_ps_0843_1993050100_123_962_12179487_0 3,366,000 18,392 0.0055
09 May 2023 15:57:57 1527792 22264399 oifs_43r3_ps_0843_1993050100_123_962_12179487_0 2,502,000 11,950 0.0048
09 May 2023 15:57:53 1527792 22264399 oifs_43r3_ps_0843_1993050100_123_962_12179487_0 1,638,000 5,824 0.0036


©2024 cpdn.org