climateprediction.net home page
Task 22289836

Task 22289836

Name oifs_43r3_ps_0740_2017050100_123_986_12203384_0
Workunit 12203384
Created 24 Dec 2022, 18:47:50 UTC
Sent 26 Dec 2022, 7:19:52 UTC
Report deadline 25 Jan 2023, 7:19:52 UTC
Received 27 Dec 2022, 12:55:01 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1524359
Run time 20 hours 19 min 13 sec
CPU time 11 hours 16 min 33 sec
Validate state Invalid
Credit 5,927.04
Device peak FLOPS 3.52 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,399.20 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,230.69 MB
Stderr 
<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
43r3_ps_12203384/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_47.zip
Uploading the intermediate file: upload_file_47.zip
  21:25:32 STEP 1153 H=1153:00 +CPU= 21.333
  21:25:56 STEP 1154 H=1154:00 +CPU= 21.299
  21:27:26 STEP 1155 H=1155:00 +CPU= 33.371
  21:27:51 STEP 1156 H=1156:00 +CPU= 21.561
  21:28:15 STEP 1157 H=1157:00 +CPU= 21.376
  21:29:30 STEP 1158 H=1158:00 +CPU= 34.061
  21:29:51 STEP 1159 H=1159:00 +CPU= 20.881
  21:30:16 STEP 1160 H=1160:00 +CPU= 21.525
  21:31:07 STEP 1161 H=1161:00 +CPU= 33.381
  21:31:29 STEP 1162 H=1162:00 +CPU= 22.392
  21:31:56 STEP 1163 H=1163:00 +CPU= 21.282
  21:33:15 STEP 1164 H=1164:00 +CPU= 37.173
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001164
  21:33:37 STEP 1165 H=1165:00 +CPU= 21.043
  21:34:11 STEP 1166 H=1166:00 +CPU= 21.727
  21:35:05 STEP 1167 H=1167:00 +CPU= 33.291
  21:35:31 STEP 1168 H=1168:00 +CPU= 21.531
  21:35:59 STEP 1169 H=1169:00 +CPU= 21.439
  21:36:59 STEP 1170 H=1170:00 +CPU= 34.181
  21:37:22 STEP 1171 H=1171:00 +CPU= 21.200
  21:37:45 STEP 1172 H=1172:00 +CPU= 21.888
  21:38:56 STEP 1173 H=1173:00 +CPU= 33.560
  21:39:18 STEP 1174 H=1174:00 +CPU= 21.488
  21:39:45 STEP 1175 H=1175:00 +CPU= 21.914
  21:40:58 STEP 1176 H=1176:00 +CPU= 39.245
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_48.zip
  21:41:43 STEP 1177 H=1177:00 +CPU= 21.578
Uploading the intermediate file: upload_file_48.zip
  21:42:04 STEP 1178 H=1178:00 +CPU= 21.394
Uploading trickle at timestep: 4233600
  21:42:57 STEP 1179 H=1179:00 +CPU= 33.622
  21:43:31 STEP 1180 H=1180:00 +CPU= 21.951
  21:43:52 STEP 1181 H=1181:00 +CPU= 20.848
  21:44:27 STEP 1182 H=1182:00 +CPU= 33.859
  21:44:48 STEP 1183 H=1183:00 +CPU= 21.111
  21:45:10 STEP 1184 H=1184:00 +CPU= 21.639
  21:45:43 STEP 1185 H=1185:00 +CPU= 33.073
  21:46:05 STEP 1186 H=1186:00 +CPU= 21.649
  21:46:26 STEP 1187 H=1187:00 +CPU= 21.002
  21:47:08 STEP 1188 H=1188:00 +CPU= 36.289
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001188
  21:47:28 STEP 1189 H=1189:00 +CPU= 20.582
  21:47:50 STEP 1190 H=1190:00 +CPU= 21.399
  21:48:22 STEP 1191 H=1191:00 +CPU= 32.665
  21:48:44 STEP 1192 H=1192:00 +CPU= 21.707
  21:49:05 STEP 1193 H=1193:00 +CPU= 20.698
  21:49:39 STEP 1194 H=1194:00 +CPU= 34.163
  21:50:00 STEP 1195 H=1195:00 +CPU= 20.977
  21:50:22 STEP 1196 H=1196:00 +CPU= 21.475
  21:50:55 STEP 1197 H=1197:00 +CPU= 32.914
  21:51:17 STEP 1198 H=1198:00 +CPU= 21.750
  21:51:40 STEP 1199 H=1199:00 +CPU= 21.848
  21:52:28 STEP 1200 H=1200:00 +CPU= 38.343
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_49.zip
Uploading the intermediate file: upload_file_49.zip
  21:52:49 STEP 1201 H=1201:00 +CPU= 21.120
  21:53:11 STEP 1202 H=1202:00 +CPU= 21.796
  21:53:44 STEP 1203 H=1203:00 +CPU= 32.698
  21:54:05 STEP 1204 H=1204:00 +CPU= 21.548
  21:54:26 STEP 1205 H=1205:00 +CPU= 20.908
  21:55:00 STEP 1206 H=1206:00 +CPU= 33.231
  21:55:20 STEP 1207 H=1207:00 +CPU= 20.738
  21:55:42 STEP 1208 H=1208:00 +CPU= 21.250
  21:56:14 STEP 1209 H=1209:00 +CPU= 32.553
  21:56:37 STEP 1210 H=1210:00 +CPU= 22.253
  21:56:57 STEP 1211 H=1211:00 +CPU= 20.581
  21:58:17 STEP 1212 H=1212:00 +CPU= 37.899
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001212
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0740
(argv4) batchid: 986
(argv5) wuid: 12203384
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2f6260bdb274eac46780234859be1fd7 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0740_2017050100_123_986_12203384.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0740_2017050100_123_986_12203384.zip
ic_ancil_file: ic_ancil_12203384
ifsdata_file: ifsdata_12203384
climate_data_file: clim_data_12203384
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6368220425fcca61853a7790c6c3a415 to: /var/lib/boinc-client/slots/3/ic_ancil_12203384.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203384.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_51be6c01313022be6e21e6261456fa2e to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203384.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203384.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203384.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203384.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203384.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203384.xml
-- Model is restarting --
Adjusting last_iter, 1213, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203384.xml
last_cpu_time: 31919
upload_file_number: 50
last_iter: 1199
last_upload: 4320000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693
The child process has been launched with process id: 13009
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:13009:13009] [20221226:221657:1672121817:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:25:20 STEP 1200 H=1200:00 +CPU= 51.038
  22:25:51 STEP 1201 H=1201:00 +CPU= 30.845
  22:26:25 STEP 1202 H=1202:00 +CPU= 32.932
  22:27:14 STEP 1203 H=1203:00 +CPU= 49.109
  22:27:46 STEP 1204 H=1204:00 +CPU= 30.995
  22:28:13 STEP 1205 H=1205:00 +CPU= 27.823
  22:28:58 STEP 1206 H=1206:00 +CPU= 44.589
  22:29:19 STEP 1207 H=1207:00 +CPU= 20.856
  22:30:05 STEP 1208 H=1208:00 +CPU= 31.501
  22:36:17 STEP 1209 H=1209:00 +CPU= 52.414
  22:37:07 STEP 1210 H=1210:00 +CPU= 34.552
  22:38:37 STEP 1211 H=1211:00 +CPU= 34.482
  22:39:45 STEP 1212 H=1212:00 +CPU= 56.674
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001212
  22:40:13 STEP 1213 H=1213:00 +CPU= 27.648
  22:40:50 STEP 1214 H=1214:00 +CPU= 36.016
  22:44:21 STEP 1215 H=1215:00 +CPU= 52.481
  22:45:04 STEP 1216 H=1216:00 +CPU= 36.640
  22:45:41 STEP 1217 H=1217:00 +CPU= 35.445
  22:48:09 STEP 1218 H=1218:00 +CPU= 55.644
  22:48:45 STEP 1219 H=1219:00 +CPU= 33.799
  22:49:22 STEP 1220 H=1220:00 +CPU= 35.511
  22:50:20 STEP 1221 H=1221:00 +CPU= 55.497
  22:50:57 STEP 1222 H=1222:00 +CPU= 36.058
  22:51:34 STEP 1223 H=1223:00 +CPU= 34.504
  22:53:05 STEP 1224 H=1224:00 +CPU= 63.958
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_50.zip
Uploading the intermediate file: upload_file_50.zip
  22:53:40 STEP 1225 H=1225:00 +CPU= 32.904
  22:54:16 STEP 1226 H=1226:00 +CPU= 35.181
  22:58:08 STEP 1227 H=1227:00 +CPU= 55.197
  22:58:41 STEP 1228 H=1228:00 +CPU= 30.929
  22:59:11 STEP 1229 H=1229:00 +CPU= 30.027
  23:02:05 STEP 1230 H=1230:00 +CPU= 55.977
  23:02:44 STEP 1231 H=1231:00 +CPU= 36.198
  23:03:22 STEP 1232 H=1232:00 +CPU= 36.948
  23:04:18 STEP 1233 H=1233:00 +CPU= 53.177
  23:04:56 STEP 1234 H=1234:00 +CPU= 36.493
  23:05:24 STEP 1235 H=1235:00 +CPU= 26.838
  23:06:19 STEP 1236 H=1236:00 +CPU= 52.437
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001236
  23:06:47 STEP 1237 H=1237:00 +CPU= 28.192
  23:07:10 STEP 1238 H=1238:00 +CPU= 22.381
  23:07:45 STEP 1239 H=1239:00 +CPU= 34.578
  23:08:08 STEP 1240 H=1240:00 +CPU= 21.486
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0740
(argv4) batchid: 986
(argv5) wuid: 12203384
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2f6260bdb274eac46780234859be1fd7 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0740_2017050100_123_986_12203384.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0740_2017050100_123_986_12203384.zip
ic_ancil_file: ic_ancil_12203384
ifsdata_file: ifsdata_12203384
climate_data_file: clim_data_12203384
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6368220425fcca61853a7790c6c3a415 to: /var/lib/boinc-client/slots/3/ic_ancil_12203384.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203384.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_51be6c01313022be6e21e6261456fa2e to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203384.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203384.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203384.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203384.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203384.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203384.xml
-- Model is restarting --
Adjusting last_iter, 1241, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203384.xml
last_cpu_time: 33536
upload_file_number: 51
last_iter: 1223
last_upload: 4406400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693
The child process has been launched with process id: 3016
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:3016:3016] [20221227:000332:1672128212:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  00:12:29 STEP 1224 H=1224:00 +CPU= 39.367
  00:16:34 STEP 1225 H=1225:00 +CPU= 21.335
  00:17:06 STEP 1226 H=1226:00 +CPU= 21.503
  00:17:51 STEP 1227 H=1227:00 +CPU= 32.961
  00:18:15 STEP 1228 H=1228:00 +CPU= 21.467
  00:18:41 STEP 1229 H=1229:00 +CPU= 21.052
  00:19:39 STEP 1230 H=1230:00 +CPU= 33.463
  00:20:07 STEP 1231 H=1231:00 +CPU= 21.027
  00:20:35 STEP 1232 H=1232:00 +CPU= 21.498
  00:21:25 STEP 1233 H=1233:00 +CPU= 33.183
  00:21:55 STEP 1234 H=1234:00 +CPU= 21.445
  00:22:24 STEP 1235 H=1235:00 +CPU= 21.019
  00:23:34 STEP 1236 H=1236:00 +CPU= 37.002
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001236
  00:24:20 STEP 1237 H=1237:00 +CPU= 21.336
  00:29:08 STEP 1238 H=1238:00 +CPU= 25.058
  00:30:18 STEP 1239 H=1239:00 +CPU= 33.939
  00:31:32 STEP 1240 H=1240:00 +CPU= 21.738
  00:31:55 STEP 1241 H=1241:00 +CPU= 20.887
  00:32:31 STEP 1242 H=1242:00 +CPU= 33.793
  00:32:53 STEP 1243 H=1243:00 +CPU= 20.741
  00:33:23 STEP 1244 H=1244:00 +CPU= 21.495
  00:34:10 STEP 1245 H=1245:00 +CPU= 32.778
  00:34:35 STEP 1246 H=1246:00 +CPU= 21.973
  00:35:06 STEP 1247 H=1247:00 +CPU= 21.738
  00:37:47 STEP 1248 H=1248:00 +CPU= 38.632
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_51.zip
Uploading the intermediate file: upload_file_51.zip
  00:38:08 STEP 1249 H=1249:00 +CPU= 21.014
  00:38:30 STEP 1250 H=1250:00 +CPU= 21.769
  00:39:03 STEP 1251 H=1251:00 +CPU= 32.481
  00:39:24 STEP 1252 H=1252:00 +CPU= 21.258
  00:39:44 STEP 1253 H=1253:00 +CPU= 20.443
  00:40:18 STEP 1254 H=1254:00 +CPU= 33.640
  00:40:39 STEP 1255 H=1255:00 +CPU= 20.799
  00:41:00 STEP 1256 H=1256:00 +CPU= 20.993
  00:41:32 STEP 1257 H=1257:00 +CPU= 32.274
  00:41:54 STEP 1258 H=1258:00 +CPU= 21.660
  00:42:14 STEP 1259 H=1259:00 +CPU= 20.390
  00:42:54 STEP 1260 H=1260:00 +CPU= 35.552
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001260
  00:43:19 STEP 1261 H=1261:00 +CPU= 20.751
  00:43:47 STEP 1262 H=1262:00 +CPU= 22.571
  00:44:22 STEP 1263 H=1263:00 +CPU= 32.484
  00:44:43 STEP 1264 H=1264:00 +CPU= 21.419
  00:45:07 STEP 1265 H=1265:00 +CPU= 20.840
  00:45:44 STEP 1266 H=1266:00 +CPU= 33.582
  00:46:05 STEP 1267 H=1267:00 +CPU= 21.040
  00:46:30 STEP 1268 H=1268:00 +CPU= 21.805
  00:47:05 STEP 1269 H=1269:00 +CPU= 33.451
  00:47:27 STEP 1270 H=1270:00 +CPU= 21.838
  00:47:51 STEP 1271 H=1271:00 +CPU= 21.660
  00:48:54 STEP 1272 H=1272:00 +CPU= 38.788
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_52.zip
Uploading the intermediate file: upload_file_52.zip
  00:49:18 STEP 1273 H=1273:00 +CPU= 21.220
  00:49:41 STEP 1274 H=1274:00 +CPU= 21.965
  00:50:17 STEP 1275 H=1275:00 +CPU= 33.333
  00:50:42 STEP 1276 H=1276:00 +CPU= 21.966
  00:51:09 STEP 1277 H=1277:00 +CPU= 21.153
  00:51:51 STEP 1278 H=1278:00 +CPU= 33.718
  00:53:24 STEP 1279 H=1279:00 +CPU= 21.053
  00:53:58 STEP 1280 H=1280:00 +CPU= 22.057
  00:55:31 STEP 1281 H=1281:00 +CPU= 33.636
  00:56:21 STEP 1282 H=1282:00 +CPU= 21.986
  00:57:09 STEP 1283 H=1283:00 +CPU= 21.247
  00:58:41 STEP 1284 H=1284:00 +CPU= 36.092
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001284
  00:59:10 STEP 1285 H=1285:00 +CPU= 20.974
  01:00:25 STEP 1286 H=1286:00 +CPU= 21.308
  01:01:15 STEP 1287 H=1287:00 +CPU= 32.339
  01:01:38 STEP 1288 H=1288:00 +CPU= 21.590
  01:02:05 STEP 1289 H=1289:00 +CPU= 20.714
  01:03:42 STEP 1290 H=1290:00 +CPU= 33.730
  01:04:34 STEP 1291 H=1291:00 +CPU= 20.848
  01:05:23 STEP 1292 H=1292:00 +CPU= 21.866
  01:07:39 STEP 1293 H=1293:00 +CPU= 33.089
  01:08:33 STEP 1294 H=1294:00 +CPU= 21.574
  01:09:01 STEP 1295 H=1295:00 +CPU= 21.692
  01:11:41 STEP 1296 H=1296:00 +CPU= 37.944
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_53.zip
Uploading the intermediate file: upload_file_53.zip
  01:13:40 STEP 1297 H=1297:00 +CPU= 21.212
  01:14:24 STEP 1298 H=1298:00 +CPU= 21.564
  01:15:19 STEP 1299 H=1299:00 +CPU= 33.235
  01:16:01 STEP 1300 H=1300:00 +CPU= 21.198
  01:16:45 STEP 1301 H=1301:00 +CPU= 20.788
  01:17:30 STEP 1302 H=1302:00 +CPU= 33.250
  01:17:55 STEP 1303 H=1303:00 +CPU= 20.997
  01:18:37 STEP 1304 H=1304:00 +CPU= 20.721
  01:19:57 STEP 1305 H=1305:00 +CPU= 32.737
  01:20:28 STEP 1306 H=1306:00 +CPU= 21.079
  01:20:51 STEP 1307 H=1307:00 +CPU= 20.403
  01:22:03 STEP 1308 H=1308:00 +CPU= 35.356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001308
  01:22:28 STEP 1309 H=1309:00 +CPU= 20.872
  01:22:55 STEP 1310 H=1310:00 +CPU= 20.652
  01:23:55 STEP 1311 H=1311:00 +CPU= 32.396
  01:24:35 STEP 1312 H=1312:00 +CPU= 21.327
  01:24:58 STEP 1313 H=1313:00 +CPU= 20.473
  01:25:58 STEP 1314 H=1314:00 +CPU= 33.347
  01:26:29 STEP 1315 H=1315:00 +CPU= 21.020
  01:26:50 STEP 1316 H=1316:00 +CPU= 21.252
  01:27:39 STEP 1317 H=1317:00 +CPU= 32.865
  01:28:04 STEP 1318 H=1318:00 +CPU= 21.082
  01:28:40 STEP 1319 H=1319:00 +CPU= 21.181
  01:30:16 STEP 1320 H=1320:00 +CPU= 38.135
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_54.zip
Uploading the intermediate file: upload_file_54.zip
  01:30:38 STEP 1321 H=1321:00 +CPU= 21.037
  01:30:59 STEP 1322 H=1322:00 +CPU= 21.021
  01:31:32 STEP 1323 H=1323:00 +CPU= 32.931
  01:31:54 STEP 1324 H=1324:00 +CPU= 21.288
  01:32:14 STEP 1325 H=1325:00 +CPU= 20.699
  01:32:50 STEP 1326 H=1326:00 +CPU= 33.878
  01:33:11 STEP 1327 H=1327:00 +CPU= 20.626
  01:33:33 STEP 1328 H=1328:00 +CPU= 21.423
  01:34:06 STEP 1329 H=1329:00 +CPU= 32.825
  01:34:27 STEP 1330 H=1330:00 +CPU= 21.209
  01:34:48 STEP 1331 H=1331:00 +CPU= 20.536
  01:35:34 STEP 1332 H=1332:00 +CPU= 35.605
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001332
  01:36:02 STEP 1333 H=1333:00 +CPU= 20.660
  01:36:24 STEP 1334 H=1334:00 +CPU= 20.870
  01:37:12 STEP 1335 H=1335:00 +CPU= 32.779
  01:37:40 STEP 1336 H=1336:00 +CPU= 21.170
  01:38:04 STEP 1337 H=1337:00 +CPU= 20.751
  01:38:40 STEP 1338 H=1338:00 +CPU= 33.048
  01:39:08 STEP 1339 H=1339:00 +CPU= 20.965
  01:39:34 STEP 1340 H=1340:00 +CPU= 21.093
  01:40:42 STEP 1341 H=1341:00 +CPU= 32.403
  01:41:04 STEP 1342 H=1342:00 +CPU= 21.160
  01:41:31 STEP 1343 H=1343:00 +CPU= 21.650
  01:43:24 STEP 1344 H=1344:00 +CPU= 37.717
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_55.zip
Uploading the intermediate file: upload_file_55.zip
  01:45:16 STEP 1345 H=1345:00 +CPU= 20.613
  01:46:22 STEP 1346 H=1346:00 +CPU= 21.643
  01:49:41 STEP 1347 H=1347:00 +CPU= 32.791
  01:50:33 STEP 1348 H=1348:00 +CPU= 21.529
  01:50:54 STEP 1349 H=1349:00 +CPU= 20.688
  01:51:28 STEP 1350 H=1350:00 +CPU= 33.086
  01:51:49 STEP 1351 H=1351:00 +CPU= 21.014
  01:52:11 STEP 1352 H=1352:00 +CPU= 21.363
  01:52:44 STEP 1353 H=1353:00 +CPU= 32.992
  01:53:05 STEP 1354 H=1354:00 +CPU= 21.712
  01:53:26 STEP 1355 H=1355:00 +CPU= 20.859
  01:54:08 STEP 1356 H=1356:00 +CPU= 36.763
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001356
  01:54:29 STEP 1357 H=1357:00 +CPU= 20.333
  01:54:50 STEP 1358 H=1358:00 +CPU= 21.133
  01:55:25 STEP 1359 H=1359:00 +CPU= 32.128
  01:55:46 STEP 1360 H=1360:00 +CPU= 21.353
  01:56:21 STEP 1361 H=1361:00 +CPU= 20.755
  01:57:02 STEP 1362 H=1362:00 +CPU= 32.939
  01:57:24 STEP 1363 H=1363:00 +CPU= 20.460
  01:57:58 STEP 1364 H=1364:00 +CPU= 21.523
  01:58:39 STEP 1365 H=1365:00 +CPU= 32.384
  01:59:07 STEP 1366 H=1366:00 +CPU= 21.240
  01:59:42 STEP 1367 H=1367:00 +CPU= 21.602
  02:06:58 STEP 1368 H=1368:00 +CPU= 38.864
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_56.zip
Uploading the intermediate file: upload_file_56.zip
  02:07:30 STEP 1369 H=1369:00 +CPU= 20.384
  02:07:56 STEP 1370 H=1370:00 +CPU= 21.178
  02:08:30 STEP 1371 H=1371:00 +CPU= 32.426
  02:08:52 STEP 1372 H=1372:00 +CPU= 21.225
  02:09:13 STEP 1373 H=1373:00 +CPU= 20.569
  02:10:22 STEP 1374 H=1374:00 +CPU= 33.431
  02:10:43 STEP 1375 H=1375:00 +CPU= 20.885
  02:11:08 STEP 1376 H=1376:00 +CPU= 21.334
  02:12:00 STEP 1377 H=1377:00 +CPU= 32.962
  02:12:22 STEP 1378 H=1378:00 +CPU= 21.159
  02:12:52 STEP 1379 H=1379:00 +CPU= 20.544
  02:13:50 STEP 1380 H=1380:00 +CPU= 35.839
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001380
  02:14:11 STEP 1381 H=1381:00 +CPU= 20.661
  02:14:32 STEP 1382 H=1382:00 +CPU= 21.213
  02:15:05 STEP 1383 H=1383:00 +CPU= 32.060
  02:15:29 STEP 1384 H=1384:00 +CPU= 21.010
  02:15:50 STEP 1385 H=1385:00 +CPU= 20.616
  02:16:22 STEP 1386 H=1386:00 +CPU= 32.472
  02:16:44 STEP 1387 H=1387:00 +CPU= 20.468
  02:17:05 STEP 1388 H=1388:00 +CPU= 20.959
  02:17:39 STEP 1389 H=1389:00 +CPU= 32.917
  02:18:00 STEP 1390 H=1390:00 +CPU= 21.117
  02:18:22 STEP 1391 H=1391:00 +CPU= 21.100
  02:19:13 STEP 1392 H=1392:00 +CPU= 37.616
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_57.zip
Uploading the intermediate file: upload_file_57.zip
  02:19:33 STEP 1393 H=1393:00 +CPU= 20.629
  02:19:55 STEP 1394 H=1394:00 +CPU= 21.599
  02:20:27 STEP 1395 H=1395:00 +CPU= 32.344
  02:20:49 STEP 1396 H=1396:00 +CPU= 21.486
  02:21:10 STEP 1397 H=1397:00 +CPU= 20.630
  02:21:43 STEP 1398 H=1398:00 +CPU= 33.119
  02:22:04 STEP 1399 H=1399:00 +CPU= 20.529
  02:22:25 STEP 1400 H=1400:00 +CPU= 21.122
  02:22:58 STEP 1401 H=1401:00 +CPU= 32.517
  02:23:19 STEP 1402 H=1402:00 +CPU= 21.000
  02:23:39 STEP 1403 H=1403:00 +CPU= 20.629
  02:24:15 STEP 1404 H=1404:00 +CPU= 35.956
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001404
  02:24:37 STEP 1405 H=1405:00 +CPU= 21.397
  02:25:00 STEP 1406 H=1406:00 +CPU= 22.019
  02:25:32 STEP 1407 H=1407:00 +CPU= 32.693
  02:25:54 STEP 1408 H=1408:00 +CPU= 21.392
  02:26:14 STEP 1409 H=1409:00 +CPU= 20.634
  02:26:48 STEP 1410 H=1410:00 +CPU= 33.402
  02:27:09 STEP 1411 H=1411:00 +CPU= 20.422
  02:27:30 STEP 1412 H=1412:00 +CPU= 21.169
  02:28:03 STEP 1413 H=1413:00 +CPU= 32.944
  02:28:25 STEP 1414 H=1414:00 +CPU= 21.129
  02:28:46 STEP 1415 H=1415:00 +CPU= 21.111
  02:29:39 STEP 1416 H=1416:00 +CPU= 37.531
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_58.zip
Uploading the intermediate file: upload_file_58.zip
  02:30:00 STEP 1417 H=1417:00 +CPU= 20.648
  02:30:25 STEP 1418 H=1418:00 +CPU= 21.413
  02:31:13 STEP 1419 H=1419:00 +CPU= 32.746
  02:32:01 STEP 1420 H=1420:00 +CPU= 22.512
  02:32:23 STEP 1421 H=1421:00 +CPU= 21.240
  02:34:22 STEP 1422 H=1422:00 +CPU= 33.442
  02:34:50 STEP 1423 H=1423:00 +CPU= 21.439
  02:35:44 STEP 1424 H=1424:00 +CPU= 21.198
  02:38:23 STEP 1425 H=1425:00 +CPU= 33.124
  02:39:01 STEP 1426 H=1426:00 +CPU= 21.326
  02:40:08 STEP 1427 H=1427:00 +CPU= 21.730
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0740
(argv4) batchid: 986
(argv5) wuid: 12203384
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/3
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2f6260bdb274eac46780234859be1fd7 to: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0740_2017050100_123_986_12203384.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/3/oifs_43r3_ps_0740_2017050100_123_986_12203384.zip
ic_ancil_file: ic_ancil_12203384
ifsdata_file: ifsdata_12203384
climate_data_file: clim_data_12203384
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_6368220425fcca61853a7790c6c3a415 to: /var/lib/boinc-client/slots/3/ic_ancil_12203384.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/3/ic_ancil_12203384.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_51be6c01313022be6e21e6261456fa2e to: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203384.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/3/ifsdata/ifsdata_12203384.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203384.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/3/159l_2/clim_data_12203384.zip
Checking for progress XML file: /var/lib/boinc-client/slots/3/progress_file_12203384.xml
Opened progress file ok : /var/lib/boinc-client/slots/3/progress_file_12203384.xml
-- Model is restarting --
Adjusting last_iter, 1428, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/3/progress_file_12203384.xml
last_cpu_time: 38768
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693
The child process has been launched with process id: 15612
Executing the command: /var/lib/boinc-client/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:torrents:1:1:15612:15612] [20221227:025804:1672138684:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  03:05:08 STEP 1416 H=1416:00 +CPU= 37.909
  03:08:42 STEP 1417 H=1417:00 +CPU= 21.056
  03:09:39 STEP 1418 H=1418:00 +CPU= 21.481
  03:10:26 STEP 1419 H=1419:00 +CPU= 32.643
  03:10:51 STEP 1420 H=1420:00 +CPU= 20.982
  03:11:19 STEP 1421 H=1421:00 +CPU= 20.924
  03:12:01 STEP 1422 H=1422:00 +CPU= 32.624
  03:12:26 STEP 1423 H=1423:00 +CPU= 20.566
  03:12:56 STEP 1424 H=1424:00 +CPU= 21.221
  03:13:35 STEP 1425 H=1425:00 +CPU= 32.533
  03:13:56 STEP 1426 H=1426:00 +CPU= 20.970
  03:14:26 STEP 1427 H=1427:00 +CPU= 20.949
  03:15:25 STEP 1428 H=1428:00 +CPU= 36.088
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001428
  03:15:49 STEP 1429 H=1429:00 +CPU= 21.258
  03:16:17 STEP 1430 H=1430:00 +CPU= 21.065
  03:17:15 STEP 1431 H=1431:00 +CPU= 33.037
  03:17:43 STEP 1432 H=1432:00 +CPU= 20.950
  03:18:04 STEP 1433 H=1433:00 +CPU= 20.332
  03:18:48 STEP 1434 H=1434:00 +CPU= 33.155
  03:19:15 STEP 1435 H=1435:00 +CPU= 20.336
  03:19:44 STEP 1436 H=1436:00 +CPU= 20.954
  03:20:22 STEP 1437 H=1437:00 +CPU= 32.428
  03:20:48 STEP 1438 H=1438:00 +CPU= 21.009
  03:21:12 STEP 1439 H=1439:00 +CPU= 21.739
  03:22:26 STEP 1440 H=1440:00 +CPU= 38.050
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_59.zip
Uploading the intermediate file: upload_file_59.zip
  03:22:55 STEP 1441 H=1441:00 +CPU= 20.352
  03:23:28 STEP 1442 H=1442:00 +CPU= 21.286
  03:24:21 STEP 1443 H=1443:00 +CPU= 32.575
  03:24:51 STEP 1444 H=1444:00 +CPU= 21.367
  03:25:12 STEP 1445 H=1445:00 +CPU= 20.763
  03:26:09 STEP 1446 H=1446:00 +CPU= 32.957
  03:26:52 STEP 1447 H=1447:00 +CPU= 21.131
  03:27:13 STEP 1448 H=1448:00 +CPU= 20.854
  03:27:58 STEP 1449 H=1449:00 +CPU= 32.295
  03:28:22 STEP 1450 H=1450:00 +CPU= 21.269
  03:28:43 STEP 1451 H=1451:00 +CPU= 20.653
  03:29:52 STEP 1452 H=1452:00 +CPU= 36.252
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001452
  03:30:34 STEP 1453 H=1453:00 +CPU= 20.578
  03:31:01 STEP 1454 H=1454:00 +CPU= 21.167
  03:31:41 STEP 1455 H=1455:00 +CPU= 36.114
  03:32:02 STEP 1456 H=1456:00 +CPU= 20.967
  03:32:22 STEP 1457 H=1457:00 +CPU= 20.730
  03:32:55 STEP 1458 H=1458:00 +CPU= 33.000
  03:33:16 STEP 1459 H=1459:00 +CPU= 20.651
  03:33:37 STEP 1460 H=1460:00 +CPU= 21.019
  03:34:10 STEP 1461 H=1461:00 +CPU= 32.503
  03:34:31 STEP 1462 H=1462:00 +CPU= 21.193
  03:34:56 STEP 1463 H=1463:00 +CPU= 21.506
  03:36:17 STEP 1464 H=1464:00 +CPU= 38.342
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12203384/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0740_2017050100_123_986_12203384_0_r1484192693_60.zip
Uploading the intermediate file: upload_file_60.zip
  03:36:43 STEP 1465 H=1465:00 +CPU= 20.489
  03:37:05 STEP 1466 H=1466:00 +CPU= 21.067
  03:37:54 STEP 1467 H=1467:00 +CPU= 32.482
  03:38:15 STEP 1468 H=1468:00 +CPU= 20.921
  03:38:40 STEP 1469 H=1469:00 +CPU= 20.512
  03:39:39 STEP 1470 H=1470:00 +CPU= 33.578
  03:40:00 STEP 1471 H=1471:00 +CPU= 20.341
  03:40:28 STEP 1472 H=1472:00 +CPU= 21.383
  03:41:08 STEP 1473 H=1473:00 +CPU= 32.542
  03:41:32 STEP 1474 H=1474:00 +CPU= 21.134
  03:42:03 STEP 1475 H=1475:00 +CPU= 20.309
  03:42:49 STEP 1476 H=1476:00 +CPU= 35.398
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/3/ICMUAhq0f+001476
  03:43:14 STEP 1477 H=1477:00 +CPU= 20.553
  03:44:38 STEP 1478 H=1478:00 +CPU= 21.555
  03:45:14 STEP 1479 H=1479:00 +CPU= 32.721
  03:45:35 STEP 1480 H=1480:00 +CPU= 21.606
  03:46:26 STEP 1481 H=1481:00 +CPU= 21.063
  03:47:23 STEP 1482 H=1482:00 +CPU= 33.099
  03:48:03 STEP 1483 H=1483:00 +CPU= 20.683
  03:48:34 STEP 1484 H=1484:00 +CPU= 21.207
  03:49:58 STEP 1485 H=1485:00 +CPU= 32.347
  03:51:00 STEP 1486 H=1486:00 +CPU= 21.753
  03:52:08 STEP 1487 H=1487:00 +CPU= 21.234
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:58:13 1524359 22289836 oifs_43r3_ps_0740_2017050100_123_986_12203384_0 4,233,600 30,990 0.0073
09 May 2023 15:57:58 1524359 22289836 oifs_43r3_ps_0740_2017050100_123_986_12203384_0 3,283,200 23,809 0.0073
09 May 2023 15:57:44 1524359 22289836 oifs_43r3_ps_0740_2017050100_123_986_12203384_0 2,419,200 17,626 0.0073
09 May 2023 15:57:25 1524359 22289836 oifs_43r3_ps_0740_2017050100_123_986_12203384_0 1,296,000 9,602 0.0074


©2024 climateprediction.net