<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
+CPU= 31.161
12:32:01 STEP 2248 H=2248:00 +CPU= 23.032
12:32:21 STEP 2249 H=2249:00 +CPU= 20.666
12:33:01 STEP 2250 H=2250:00 +CPU= 32.779
12:33:39 STEP 2251 H=2251:00 +CPU= 25.160
12:34:22 STEP 2252 H=2252:00 +CPU= 26.051
12:35:25 STEP 2253 H=2253:00 +CPU= 38.169
12:35:55 STEP 2254 H=2254:00 +CPU= 24.358
12:36:24 STEP 2255 H=2255:00 +CPU= 24.849
12:37:15 STEP 2256 H=2256:00 +CPU= 42.426
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002256
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002232
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002244
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002244
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917_93.zip
Uploading the intermediate file: upload_file_93.zip
12:37:41 STEP 2257 H=2257:00 +CPU= 21.002
12:38:05 STEP 2258 H=2258:00 +CPU= 20.377
12:38:43 STEP 2259 H=2259:00 +CPU= 31.244
12:39:08 STEP 2260 H=2260:00 +CPU= 21.607
12:39:31 STEP 2261 H=2261:00 +CPU= 19.439
12:40:08 STEP 2262 H=2262:00 +CPU= 32.269
12:40:33 STEP 2263 H=2263:00 +CPU= 20.463
12:40:59 STEP 2264 H=2264:00 +CPU= 21.160
12:41:39 STEP 2265 H=2265:00 +CPU= 33.125
12:42:05 STEP 2266 H=2266:00 +CPU= 22.145
12:42:31 STEP 2267 H=2267:00 +CPU= 21.216
12:43:13 STEP 2268 H=2268:00 +CPU= 36.116
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002268
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002268
12:43:38 STEP 2269 H=2269:00 +CPU= 21.423
12:44:04 STEP 2270 H=2270:00 +CPU= 21.794
12:44:44 STEP 2271 H=2271:00 +CPU= 33.259
12:45:11 STEP 2272 H=2272:00 +CPU= 21.931
12:45:36 STEP 2273 H=2273:00 +CPU= 21.115
12:46:17 STEP 2274 H=2274:00 +CPU= 33.713
12:46:42 STEP 2275 H=2275:00 +CPU= 21.508
12:47:08 STEP 2276 H=2276:00 +CPU= 21.637
12:47:47 STEP 2277 H=2277:00 +CPU= 33.287
12:48:13 STEP 2278 H=2278:00 +CPU= 21.988
12:48:39 STEP 2279 H=2279:00 +CPU= 21.932
12:49:25 STEP 2280 H=2280:00 +CPU= 38.820
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002280
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002256
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002268
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002268
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917_94.zip
Uploading the intermediate file: upload_file_94.zip
12:49:51 STEP 2281 H=2281:00 +CPU= 21.806
12:50:17 STEP 2282 H=2282:00 +CPU= 21.826
12:50:55 STEP 2283 H=2283:00 +CPU= 33.270
12:51:16 STEP 2284 H=2284:00 +CPU= 20.403
12:51:36 STEP 2285 H=2285:00 +CPU= 19.850
12:52:07 STEP 2286 H=2286:00 +CPU= 31.210
12:52:26 STEP 2287 H=2287:00 +CPU= 19.006
12:52:47 STEP 2288 H=2288:00 +CPU= 19.846
12:53:20 STEP 2289 H=2289:00 +CPU= 31.976
12:53:42 STEP 2290 H=2290:00 +CPU= 20.929
12:54:04 STEP 2291 H=2291:00 +CPU= 20.673
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
12:55:27 STEP 2292 H=2292:00 +CPU= 36.201
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002292
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002292
12:55:48 STEP 2293 H=2293:00 +CPU= 19.643
12:56:09 STEP 2294 H=2294:00 +CPU= 20.497
12:56:42 STEP 2295 H=2295:00 +CPU= 31.478
12:57:04 STEP 2296 H=2296:00 +CPU= 21.682
12:57:25 STEP 2297 H=2297:00 +CPU= 20.528
12:57:58 STEP 2298 H=2298:00 +CPU= 32.078
12:58:18 STEP 2299 H=2299:00 +CPU= 20.334
12:58:39 STEP 2300 H=2300:00 +CPU= 20.340
12:59:10 STEP 2301 H=2301:00 +CPU= 31.067
12:59:31 STEP 2302 H=2302:00 +CPU= 20.850
12:59:53 STEP 2303 H=2303:00 +CPU= 21.472
13:00:30 STEP 2304 H=2304:00 +CPU= 36.524
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002304
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002280
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002292
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002292
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917_95.zip
Uploading the intermediate file: upload_file_95.zip
13:00:50 STEP 2305 H=2305:00 +CPU= 19.866
13:01:09 STEP 2306 H=2306:00 +CPU= 19.285
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4749MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4749 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.429] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALL_SL_HEAP
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CALL_SL
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-CALL_SL 1 (1004)
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LAPINEA
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130125:1673438485:7143.430] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : LARMES
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130126:1673438486:7144.430] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:3155:3155] [20230111:130126:1673438486:7144.430] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000DE9516 larmes_ 484 larmes.F90
oifs_43r3_model.e 0000000000DE50FD lapinea_ 270 lapinea.F90
oifs_43r3_model.e 0000000000D7A4EC call_sl_ 300 call_sl.F90
oifs_43r3_model.e 0000000000D784FD call_sl_heap_ 94 call_sl_heap.F90
oifs_43r3_model.e 0000000000BF9958 gp_model_ 533 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0923
(argv4) batchid: 988
(argv5) wuid: 12205567
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_09b2afe92d9c7c9e1d3a6e7b53187278 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0923_2019050100_123_988_12205567.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0923_2019050100_123_988_12205567.zip
ic_ancil_file: ic_ancil_12205567
ifsdata_file: ifsdata_12205567
climate_data_file: clim_data_12205567
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e0d31a8705e2d0b5be4a8c2927b59024 to: /var/lib/boinc-client/slots/0/ic_ancil_12205567.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12205567.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0008be2fbee534233bf6cd639b3a00d5 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205567.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205567.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205567.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205567.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12205567.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12205567.xml
-- Model is restarting --
Adjusting last_iter, 2307, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12205567.xml
last_cpu_time: 62694
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917
The child process has been launched with process id: 2827
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130236:1673438556:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
13:03:18 STEP 2304 H=2304:00 +CPU= 36.413
13:03:37 STEP 2305 H=2305:00 +CPU= 18.455
13:03:58 STEP 2306 H=2306:00 +CPU= 21.168
13:04:29 STEP 2307 H=2307:00 +CPU= 30.526
13:04:50 STEP 2308 H=2308:00 +CPU= 20.874
13:05:10 STEP 2309 H=2309:00 +CPU= 19.702
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4032MB (heap), 4403MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4032 MB (maxheap), 4403 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADINTG
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADINTG-RADLSW (1210)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADIATION_SCHEME
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_interface:radiation
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130515:1673438715:159.103] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_mcica_sw:solver_mcica_sw
double free or corruption (top)
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130516:1673438716:160.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2827:2827] [20230111:130516:1673438716:160.103] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001CC9191 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001CC0C9D Unknown Unknown Unknown
oifs_43r3_model.e 000000000188C190 radiation_two_str 384 radiation_two_stream.F90
oifs_43r3_model.e 000000000196A573 radiation_mcica_s 259 radiation_mcica_sw.F90
oifs_43r3_model.e 0000000001862DC8 radiation_interfa 419 radiation_interface.F90
oifs_43r3_model.e 00000000012D30E5 radiation_scheme_ 484 radiation_scheme.F90
oifs_43r3_model.e 000000000129A415 radintg_ 1455 radintg.F90
oifs_43r3_model.e 0000000001270124 raddrv_ 537 raddrv.F90
oifs_43r3_model.e 0000000000BFD839 gp_model_ 482 gp_model.F90
oifs_43r3_model.e 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BBC1CE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 000000000057A664 stepo_ 327 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0923
(argv4) batchid: 988
(argv5) wuid: 12205567
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_09b2afe92d9c7c9e1d3a6e7b53187278 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0923_2019050100_123_988_12205567.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0923_2019050100_123_988_12205567.zip
ic_ancil_file: ic_ancil_12205567
ifsdata_file: ifsdata_12205567
climate_data_file: clim_data_12205567
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e0d31a8705e2d0b5be4a8c2927b59024 to: /var/lib/boinc-client/slots/0/ic_ancil_12205567.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12205567.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0008be2fbee534233bf6cd639b3a00d5 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205567.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205567.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205567.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205567.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12205567.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12205567.xml
-- Model is restarting --
Adjusting last_iter, 2309, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12205567.xml
last_cpu_time: 62838
upload_file_number: 96
last_iter: 2303
last_upload: 8294400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917
The child process has been launched with process id: 2912
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:215140:1673470300:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:52:43 STEP 2304 H=2304:00 +CPU= 34.454
21:53:03 STEP 2305 H=2305:00 +CPU= 19.125
21:53:24 STEP 2306 H=2306:00 +CPU= 20.449
21:53:54 STEP 2307 H=2307:00 +CPU= 29.521
21:54:14 STEP 2308 H=2308:00 +CPU= 19.632
21:54:33 STEP 2309 H=2309:00 +CPU= 18.855
21:55:03 STEP 2310 H=2310:00 +CPU= 30.224
21:55:22 STEP 2311 H=2311:00 +CPU= 19.090
21:55:42 STEP 2312 H=2312:00 +CPU= 19.743
21:56:12 STEP 2313 H=2313:00 +CPU= 29.382
21:56:32 STEP 2314 H=2314:00 +CPU= 19.334
21:56:51 STEP 2315 H=2315:00 +CPU= 18.948
21:57:23 STEP 2316 H=2316:00 +CPU= 31.937
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002316
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002316
21:57:42 STEP 2317 H=2317:00 +CPU= 18.818
21:58:02 STEP 2318 H=2318:00 +CPU= 20.062
21:58:34 STEP 2319 H=2319:00 +CPU= 30.493
21:58:54 STEP 2320 H=2320:00 +CPU= 19.973
21:59:14 STEP 2321 H=2321:00 +CPU= 19.714
21:59:44 STEP 2322 H=2322:00 +CPU= 29.238
22:00:03 STEP 2323 H=2323:00 +CPU= 19.336
22:00:23 STEP 2324 H=2324:00 +CPU= 19.383
22:00:52 STEP 2325 H=2325:00 +CPU= 29.135
22:01:12 STEP 2326 H=2326:00 +CPU= 19.609
22:01:32 STEP 2327 H=2327:00 +CPU= 20.322
22:02:09 STEP 2328 H=2328:00 +CPU= 36.529
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002328
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002304
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002316
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002316
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917_96.zip
Uploading the intermediate file: upload_file_96.zip
22:02:32 STEP 2329 H=2329:00 +CPU= 21.454
22:02:53 STEP 2330 H=2330:00 +CPU= 20.602
22:03:22 STEP 2331 H=2331:00 +CPU= 28.842
22:03:41 STEP 2332 H=2332:00 +CPU= 18.988
22:04:00 STEP 2333 H=2333:00 +CPU= 19.197
22:04:30 STEP 2334 H=2334:00 +CPU= 29.605
22:04:49 STEP 2335 H=2335:00 +CPU= 19.115
22:05:08 STEP 2336 H=2336:00 +CPU= 18.677
22:05:37 STEP 2337 H=2337:00 +CPU= 29.582
22:05:57 STEP 2338 H=2338:00 +CPU= 19.766
22:06:18 STEP 2339 H=2339:00 +CPU= 20.337
22:06:51 STEP 2340 H=2340:00 +CPU= 32.823
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002340
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002340
22:07:10 STEP 2341 H=2341:00 +CPU= 18.819
22:07:30 STEP 2342 H=2342:00 +CPU= 20.140
22:08:00 STEP 2343 H=2343:00 +CPU= 29.547
22:08:20 STEP 2344 H=2344:00 +CPU= 19.803
22:08:39 STEP 2345 H=2345:00 +CPU= 19.235
22:09:08 STEP 2346 H=2346:00 +CPU= 28.793
22:09:27 STEP 2347 H=2347:00 +CPU= 18.932
22:09:47 STEP 2348 H=2348:00 +CPU= 19.438
22:10:16 STEP 2349 H=2349:00 +CPU= 28.855
22:10:36 STEP 2350 H=2350:00 +CPU= 19.310
22:10:55 STEP 2351 H=2351:00 +CPU= 19.666
22:11:29 STEP 2352 H=2352:00 +CPU= 33.562
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002352
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002328
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002340
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002340
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917_97.zip
Uploading the intermediate file: upload_file_97.zip
22:11:48 STEP 2353 H=2353:00 +CPU= 18.627
22:12:08 STEP 2354 H=2354:00 +CPU= 20.005
22:12:38 STEP 2355 H=2355:00 +CPU= 29.329
22:12:57 STEP 2356 H=2356:00 +CPU= 19.479
22:13:18 STEP 2357 H=2357:00 +CPU= 19.937
22:13:50 STEP 2358 H=2358:00 +CPU= 31.954
22:14:10 STEP 2359 H=2359:00 +CPU= 19.643
22:14:32 STEP 2360 H=2360:00 +CPU= 20.786
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.112] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2596MB (heap), 4419MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2598 MB (maxheap), 4419 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SPCM
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SPCSI
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-SPCSI LDONEM=F SITNU (1657)
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SITNU
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VERINT
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221501:1673471701:1400.113] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : VERINT_DGEMM_2
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221502:1673471702:1401.113] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:philip-All-Series:1:1:2912:2912] [20230111:221502:1673471702:1401.113] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CE916B Unknown Unknown Unknown
oifs_43r3_model.e 000000000081FFF1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DC9090 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001711787 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000A3E48D verint_ 168 verint.F90
oifs_43r3_model.e 0000000000CE5158 sitnu_ 133 sitnu.F90
oifs_43r3_model.e 000000000132D158 spcsi_ 367 spcsi.F90
oifs_43r3_model.e 00000000012F5708 spcm_ 269 spcm.F90
oifs_43r3_model.e 000000000057A965 stepo_ 405 stepo.F90
oifs_43r3_model.e 000000000055DEEC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000005471C9 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 0000000000546412 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 0000000000545E28 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 000000000040708D cnt0_ 146 cnt0.F90
oifs_43r3_model.e 000000000040220F MAIN__ 96 master.F90
oifs_43r3_model.e 00000000004021A2 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCA390 Unknown Unknown Unknown
oifs_43r3_model.e 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2019050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0923
(argv4) batchid: 988
(argv5) wuid: 12205567
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/0
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_09b2afe92d9c7c9e1d3a6e7b53187278 to: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0923_2019050100_123_988_12205567.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/0/oifs_43r3_ps_0923_2019050100_123_988_12205567.zip
ic_ancil_file: ic_ancil_12205567
ifsdata_file: ifsdata_12205567
climate_data_file: clim_data_12205567
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_e0d31a8705e2d0b5be4a8c2927b59024 to: /var/lib/boinc-client/slots/0/ic_ancil_12205567.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/0/ic_ancil_12205567.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_0008be2fbee534233bf6cd639b3a00d5 to: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205567.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/0/ifsdata/ifsdata_12205567.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205567.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/0/159l_2/clim_data_12205567.zip
Checking for progress XML file: /var/lib/boinc-client/slots/0/progress_file_12205567.xml
Opened progress file ok : /var/lib/boinc-client/slots/0/progress_file_12205567.xml
-- Model is restarting --
Adjusting last_iter, 2361, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/0/progress_file_12205567.xml
last_cpu_time: 64191
upload_file_number: 98
last_iter: 2351
last_upload: 8467200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917
The child process has been launched with process id: 2898
Executing the command: /var/lib/boinc-client/slots/0/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:philip-All-Series:1:1:2898:2898] [20230111:221906:1673471946:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:20:20 STEP 2352 H=2352:00 +CPU= 34.951
22:20:39 STEP 2353 H=2353:00 +CPU= 19.098
22:21:00 STEP 2354 H=2354:00 +CPU= 19.718
22:21:30 STEP 2355 H=2355:00 +CPU= 29.459
22:21:49 STEP 2356 H=2356:00 +CPU= 19.438
22:22:10 STEP 2357 H=2357:00 +CPU= 19.733
22:22:41 STEP 2358 H=2358:00 +CPU= 30.195
22:23:01 STEP 2359 H=2359:00 +CPU= 19.725
22:23:21 STEP 2360 H=2360:00 +CPU= 20.072
22:23:51 STEP 2361 H=2361:00 +CPU= 29.293
22:24:12 STEP 2362 H=2362:00 +CPU= 19.943
22:24:32 STEP 2363 H=2363:00 +CPU= 19.341
22:25:04 STEP 2364 H=2364:00 +CPU= 31.575
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002364
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002364
22:25:23 STEP 2365 H=2365:00 +CPU= 19.601
22:25:44 STEP 2366 H=2366:00 +CPU= 19.648
22:26:15 STEP 2367 H=2367:00 +CPU= 30.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:26:49 STEP 2368 H=2368:00 +CPU= 21.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:27:23 STEP 2369 H=2369:00 +CPU= 21.054
22:27:58 STEP 2370 H=2370:00 +CPU= 32.327
22:28:19 STEP 2371 H=2371:00 +CPU= 20.306
22:28:42 STEP 2372 H=2372:00 +CPU= 21.554
22:29:16 STEP 2373 H=2373:00 +CPU= 32.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
22:29:52 STEP 2374 H=2374:00 +CPU= 21.747
22:30:15 STEP 2375 H=2375:00 +CPU= 21.822
22:30:54 STEP 2376 H=2376:00 +CPU= 36.805
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMGGhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMSHhq0f+002376
Moving to projects directory: /var/lib/boinc-client/slots/0/ICMUAhq0f+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002352
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMGGhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMSHhq0f+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12205567/ICMUAhq0f+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0923_2019050100_123_988_12205567_0_r651864917_98.zip
Uploading the intermediate file: upload_file_98.zip
22:31:16 STEP 2377 H=2377:00 +CPU= 21.270
22:31:39 STEP 2378 H=2378:00 +CPU= 21.581
22:32:14 STEP 2379 H=2379:00 +CPU= 32.049
22:32:38 STEP 2380 H=2380:00 +CPU= 22.557
22:33:02 STEP 2381 H=2381:00 +CPU= 21.407
22:33:37 STEP 2382 H=2382:00 +CPU= 32.905
22:33:59 STEP 2383 H=2383:00 +CPU= 20.985
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
