climateprediction.net home page
Task 22303610

Task 22303610

Name oifs_43r3_ps_0114_2009050100_123_978_12194758_1
Workunit 12194758
Created 25 Jan 2023, 18:10:28 UTC
Sent 25 Jan 2023, 18:12:37 UTC
Report deadline 26 Mar 2023, 18:12:37 UTC
Received 26 Jan 2023, 7:48:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532237
Run time 11 hours 37 min 41 sec
CPU time 11 hours 26 min 14 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,408.79 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.35 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
  20:15:50 STEP 1285 H=1285:00 +CPU= 20.363
  20:16:09 STEP 1286 H=1286:00 +CPU= 19.337
  20:16:42 STEP 1287 H=1287:00 +CPU= 32.050
  20:17:06 STEP 1288 H=1288:00 +CPU= 23.722
  20:17:28 STEP 1289 H=1289:00 +CPU= 22.530
  20:18:01 STEP 1290 H=1290:00 +CPU= 32.093
  20:18:23 STEP 1291 H=1291:00 +CPU= 21.706
  20:18:46 STEP 1292 H=1292:00 +CPU= 22.985
  20:19:19 STEP 1293 H=1293:00 +CPU= 32.282
  20:19:41 STEP 1294 H=1294:00 +CPU= 22.503
  20:20:05 STEP 1295 H=1295:00 +CPU= 23.452
  20:20:44 STEP 1296 H=1296:00 +CPU= 39.154
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_53.zip
Uploading the intermediate file: upload_file_53.zip
  20:21:05 STEP 1297 H=1297:00 +CPU= 20.738
  20:21:26 STEP 1298 H=1298:00 +CPU= 20.575
  20:21:59 STEP 1299 H=1299:00 +CPU= 32.799
  20:22:22 STEP 1300 H=1300:00 +CPU= 23.066
  20:22:43 STEP 1301 H=1301:00 +CPU= 20.873
  20:23:16 STEP 1302 H=1302:00 +CPU= 32.365
  20:23:39 STEP 1303 H=1303:00 +CPU= 22.508
  20:24:01 STEP 1304 H=1304:00 +CPU= 22.399
  20:24:33 STEP 1305 H=1305:00 +CPU= 31.481
  20:24:55 STEP 1306 H=1306:00 +CPU= 22.184
  20:25:16 STEP 1307 H=1307:00 +CPU= 20.449
  20:25:53 STEP 1308 H=1308:00 +CPU= 36.582
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001308
  20:26:16 STEP 1309 H=1309:00 +CPU= 22.675
  20:26:37 STEP 1310 H=1310:00 +CPU= 21.217
  20:27:09 STEP 1311 H=1311:00 +CPU= 31.649
  20:27:32 STEP 1312 H=1312:00 +CPU= 22.433
  20:27:52 STEP 1313 H=1313:00 +CPU= 19.980
  20:28:23 STEP 1314 H=1314:00 +CPU= 30.949
  20:28:46 STEP 1315 H=1315:00 +CPU= 22.125
  20:29:07 STEP 1316 H=1316:00 +CPU= 21.066
  20:29:37 STEP 1317 H=1317:00 +CPU= 29.697
  20:29:59 STEP 1318 H=1318:00 +CPU= 22.386
  20:30:24 STEP 1319 H=1319:00 +CPU= 24.025
  20:31:02 STEP 1320 H=1320:00 +CPU= 37.844
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_54.zip
Uploading the intermediate file: upload_file_54.zip
  20:31:23 STEP 1321 H=1321:00 +CPU= 20.992
  20:31:44 STEP 1322 H=1322:00 +CPU= 21.146
  20:32:15 STEP 1323 H=1323:00 +CPU= 30.836
  20:32:38 STEP 1324 H=1324:00 +CPU= 22.288
  20:33:00 STEP 1325 H=1325:00 +CPU= 21.656
  20:33:33 STEP 1326 H=1326:00 +CPU= 33.349
  20:33:55 STEP 1327 H=1327:00 +CPU= 21.930
  20:34:19 STEP 1328 H=1328:00 +CPU= 23.426
  20:34:52 STEP 1329 H=1329:00 +CPU= 32.786
  20:35:12 STEP 1330 H=1330:00 +CPU= 20.094
  20:35:32 STEP 1331 H=1331:00 +CPU= 20.194
  20:36:09 STEP 1332 H=1332:00 +CPU= 36.282
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001332
  20:36:31 STEP 1333 H=1333:00 +CPU= 21.886
  20:36:52 STEP 1334 H=1334:00 +CPU= 21.230
  20:37:25 STEP 1335 H=1335:00 +CPU= 32.082
  20:37:48 STEP 1336 H=1336:00 +CPU= 22.826
  20:38:11 STEP 1337 H=1337:00 +CPU= 22.985
  20:38:43 STEP 1338 H=1338:00 +CPU= 31.514
  20:39:02 STEP 1339 H=1339:00 +CPU= 19.303
  20:39:26 STEP 1340 H=1340:00 +CPU= 23.710
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0114
(argv4) batchid: 978
(argv5) wuid: 12194758
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_44ae5f80e9496dd8778b9bea20bea134 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0114_2009050100_123_978_12194758.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0114_2009050100_123_978_12194758.zip
ic_ancil_file: ic_ancil_12194758
ifsdata_file: ifsdata_12194758
climate_data_file: clim_data_12194758
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_caff69d5e7c798bae75b5ac4a0228f16 to: /var/lib/boinc-client/slots/8/ic_ancil_12194758.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12194758.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f378b9847cd8695263ee8b3e30f00712 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12194758.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12194758.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12194758.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12194758.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12194758.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12194758.xml
-- Model is restarting --
Adjusting last_iter, 1341, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12194758.xml
last_cpu_time: 33132
upload_file_number: 55
last_iter: 1319
last_upload: 4752000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770
The child process has been launched with process id: 3928581
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928581:3928581] [20230125:204103:1674704463:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:41:43 STEP 1320 H=1320:00 +CPU= 36.553
  20:42:03 STEP 1321 H=1321:00 +CPU= 19.782
  20:42:25 STEP 1322 H=1322:00 +CPU= 21.729
  20:42:57 STEP 1323 H=1323:00 +CPU= 31.702
  20:43:19 STEP 1324 H=1324:00 +CPU= 22.074
  20:43:40 STEP 1325 H=1325:00 +CPU= 20.756
  20:44:15 STEP 1326 H=1326:00 +CPU= 34.230
  20:44:36 STEP 1327 H=1327:00 +CPU= 21.137
  20:44:58 STEP 1328 H=1328:00 +CPU= 21.749
  20:45:31 STEP 1329 H=1329:00 +CPU= 33.232
  20:45:54 STEP 1330 H=1330:00 +CPU= 22.966
  20:46:16 STEP 1331 H=1331:00 +CPU= 21.785
  20:46:50 STEP 1332 H=1332:00 +CPU= 33.644
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001332
  20:47:12 STEP 1333 H=1333:00 +CPU= 21.504
  20:47:34 STEP 1334 H=1334:00 +CPU= 21.797
  20:48:05 STEP 1335 H=1335:00 +CPU= 31.288
  20:48:29 STEP 1336 H=1336:00 +CPU= 23.740
  20:48:54 STEP 1337 H=1337:00 +CPU= 24.269
  20:49:26 STEP 1338 H=1338:00 +CPU= 32.168
  20:49:46 STEP 1339 H=1339:00 +CPU= 20.184
  20:50:10 STEP 1340 H=1340:00 +CPU= 22.882
  20:50:42 STEP 1341 H=1341:00 +CPU= 31.881
  20:51:02 STEP 1342 H=1342:00 +CPU= 20.575
  20:51:25 STEP 1343 H=1343:00 +CPU= 22.271
  20:52:02 STEP 1344 H=1344:00 +CPU= 36.693
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_55.zip
Uploading the intermediate file: upload_file_55.zip
  20:52:24 STEP 1345 H=1345:00 +CPU= 22.245
  20:52:45 STEP 1346 H=1346:00 +CPU= 20.778
  20:53:21 STEP 1347 H=1347:00 +CPU= 35.057
  20:53:44 STEP 1348 H=1348:00 +CPU= 23.634
  20:54:04 STEP 1349 H=1349:00 +CPU= 19.372
  20:54:36 STEP 1350 H=1350:00 +CPU= 31.411
  20:54:59 STEP 1351 H=1351:00 +CPU= 23.490
  20:55:21 STEP 1352 H=1352:00 +CPU= 21.383
  20:55:50 STEP 1353 H=1353:00 +CPU= 29.292
  20:56:12 STEP 1354 H=1354:00 +CPU= 21.966
  20:56:35 STEP 1355 H=1355:00 +CPU= 22.710
  20:57:10 STEP 1356 H=1356:00 +CPU= 34.653
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001356
  20:57:31 STEP 1357 H=1357:00 +CPU= 20.812
  20:57:54 STEP 1358 H=1358:00 +CPU= 22.025
  20:58:26 STEP 1359 H=1359:00 +CPU= 31.815
  20:58:48 STEP 1360 H=1360:00 +CPU= 21.669
  20:59:09 STEP 1361 H=1361:00 +CPU= 21.543
  20:59:44 STEP 1362 H=1362:00 +CPU= 34.345
  21:00:04 STEP 1363 H=1363:00 +CPU= 19.944
  21:00:27 STEP 1364 H=1364:00 +CPU= 22.154
  21:01:01 STEP 1365 H=1365:00 +CPU= 34.172
  21:01:21 STEP 1366 H=1366:00 +CPU= 20.316
  21:01:42 STEP 1367 H=1367:00 +CPU= 20.542
  21:02:22 STEP 1368 H=1368:00 +CPU= 39.055
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_56.zip
Uploading the intermediate file: upload_file_56.zip
  21:02:43 STEP 1369 H=1369:00 +CPU= 21.562
  21:03:03 STEP 1370 H=1370:00 +CPU= 19.812
  21:03:37 STEP 1371 H=1371:00 +CPU= 33.641
  21:03:59 STEP 1372 H=1372:00 +CPU= 21.811
  21:04:18 STEP 1373 H=1373:00 +CPU= 18.767
  21:04:52 STEP 1374 H=1374:00 +CPU= 33.028
  21:05:15 STEP 1375 H=1375:00 +CPU= 23.146
  21:05:36 STEP 1376 H=1376:00 +CPU= 21.480
  21:06:07 STEP 1377 H=1377:00 +CPU= 29.984
  21:06:30 STEP 1378 H=1378:00 +CPU= 23.059
  21:06:53 STEP 1379 H=1379:00 +CPU= 22.458
  21:07:26 STEP 1380 H=1380:00 +CPU= 32.750
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001380
  21:07:49 STEP 1381 H=1381:00 +CPU= 22.870
  21:08:12 STEP 1382 H=1382:00 +CPU= 22.905
  21:08:41 STEP 1383 H=1383:00 +CPU= 29.161
  21:09:02 STEP 1384 H=1384:00 +CPU= 20.798
  21:09:24 STEP 1385 H=1385:00 +CPU= 21.792
  21:09:57 STEP 1386 H=1386:00 +CPU= 32.062
  21:10:17 STEP 1387 H=1387:00 +CPU= 19.791
  21:10:38 STEP 1388 H=1388:00 +CPU= 21.177
  21:11:10 STEP 1389 H=1389:00 +CPU= 31.415
  21:11:31 STEP 1390 H=1390:00 +CPU= 21.475
  21:11:53 STEP 1391 H=1391:00 +CPU= 21.756
  21:12:33 STEP 1392 H=1392:00 +CPU= 39.788
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_57.zip
Uploading the intermediate file: upload_file_57.zip
  21:12:56 STEP 1393 H=1393:00 +CPU= 22.548
  21:13:18 STEP 1394 H=1394:00 +CPU= 21.308
  21:13:50 STEP 1395 H=1395:00 +CPU= 32.535
  21:14:12 STEP 1396 H=1396:00 +CPU= 21.088
  21:14:32 STEP 1397 H=1397:00 +CPU= 20.708
  21:15:05 STEP 1398 H=1398:00 +CPU= 32.459
  21:15:28 STEP 1399 H=1399:00 +CPU= 22.204
  21:15:51 STEP 1400 H=1400:00 +CPU= 23.121
  21:16:22 STEP 1401 H=1401:00 +CPU= 30.470
  21:16:43 STEP 1402 H=1402:00 +CPU= 21.607
  21:17:06 STEP 1403 H=1403:00 +CPU= 22.049
  21:17:43 STEP 1404 H=1404:00 +CPU= 36.999
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001404
  21:18:05 STEP 1405 H=1405:00 +CPU= 21.714
  21:18:26 STEP 1406 H=1406:00 +CPU= 20.960
  21:19:00 STEP 1407 H=1407:00 +CPU= 33.507
  21:19:22 STEP 1408 H=1408:00 +CPU= 22.232
  21:19:42 STEP 1409 H=1409:00 +CPU= 19.585
  21:20:17 STEP 1410 H=1410:00 +CPU= 34.089
  21:20:40 STEP 1411 H=1411:00 +CPU= 23.118
  21:21:01 STEP 1412 H=1412:00 +CPU= 20.278
  21:21:32 STEP 1413 H=1413:00 +CPU= 31.146
  21:21:55 STEP 1414 H=1414:00 +CPU= 22.662
  21:22:18 STEP 1415 H=1415:00 +CPU= 22.769
  21:22:54 STEP 1416 H=1416:00 +CPU= 35.773
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_58.zip
Uploading the intermediate file: upload_file_58.zip
  21:23:14 STEP 1417 H=1417:00 +CPU= 20.009
  21:23:33 STEP 1418 H=1418:00 +CPU= 19.522
  21:24:07 STEP 1419 H=1419:00 +CPU= 33.413
  21:24:29 STEP 1420 H=1420:00 +CPU= 21.253
  21:24:48 STEP 1421 H=1421:00 +CPU= 19.788
  21:25:23 STEP 1422 H=1422:00 +CPU= 33.620
  21:25:46 STEP 1423 H=1423:00 +CPU= 22.873
  21:26:07 STEP 1424 H=1424:00 +CPU= 21.728
  21:26:37 STEP 1425 H=1425:00 +CPU= 29.681
  21:27:01 STEP 1426 H=1426:00 +CPU= 23.875
  21:27:26 STEP 1427 H=1427:00 +CPU= 24.014
  21:28:01 STEP 1428 H=1428:00 +CPU= 35.072
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001428
  21:28:24 STEP 1429 H=1429:00 +CPU= 22.378
  21:28:48 STEP 1430 H=1430:00 +CPU= 23.688
  21:29:20 STEP 1431 H=1431:00 +CPU= 31.891
  21:29:41 STEP 1432 H=1432:00 +CPU= 20.905
  21:30:03 STEP 1433 H=1433:00 +CPU= 22.096
  21:30:38 STEP 1434 H=1434:00 +CPU= 34.644
  21:30:58 STEP 1435 H=1435:00 +CPU= 20.061
  21:31:20 STEP 1436 H=1436:00 +CPU= 21.079
  21:31:56 STEP 1437 H=1437:00 +CPU= 35.887
  21:32:18 STEP 1438 H=1438:00 +CPU= 21.994
  21:32:38 STEP 1439 H=1439:00 +CPU= 20.040
  21:33:19 STEP 1440 H=1440:00 +CPU= 40.220
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_59.zip
Uploading the intermediate file: upload_file_59.zip
  21:33:41 STEP 1441 H=1441:00 +CPU= 21.699
  21:34:02 STEP 1442 H=1442:00 +CPU= 21.136
  21:34:37 STEP 1443 H=1443:00 +CPU= 33.770
  21:35:01 STEP 1444 H=1444:00 +CPU= 23.878
  21:35:21 STEP 1445 H=1445:00 +CPU= 20.108
  21:35:53 STEP 1446 H=1446:00 +CPU= 31.070
  21:36:15 STEP 1447 H=1447:00 +CPU= 22.503
  21:36:36 STEP 1448 H=1448:00 +CPU= 20.937
  21:37:06 STEP 1449 H=1449:00 +CPU= 29.420
  21:37:29 STEP 1450 H=1450:00 +CPU= 23.178
  21:37:54 STEP 1451 H=1451:00 +CPU= 23.674
  21:38:27 STEP 1452 H=1452:00 +CPU= 33.701
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001452
  21:38:48 STEP 1453 H=1453:00 +CPU= 20.522
  21:39:12 STEP 1454 H=1454:00 +CPU= 23.624
  21:39:46 STEP 1455 H=1455:00 +CPU= 34.218
  21:40:09 STEP 1456 H=1456:00 +CPU= 22.452
  21:40:30 STEP 1457 H=1457:00 +CPU= 21.068
  21:41:03 STEP 1458 H=1458:00 +CPU= 32.832
  21:41:26 STEP 1459 H=1459:00 +CPU= 22.030
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0114
(argv4) batchid: 978
(argv5) wuid: 12194758
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_44ae5f80e9496dd8778b9bea20bea134 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0114_2009050100_123_978_12194758.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0114_2009050100_123_978_12194758.zip
ic_ancil_file: ic_ancil_12194758
ifsdata_file: ifsdata_12194758
climate_data_file: clim_data_12194758
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_caff69d5e7c798bae75b5ac4a0228f16 to: /var/lib/boinc-client/slots/8/ic_ancil_12194758.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12194758.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f378b9847cd8695263ee8b3e30f00712 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12194758.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12194758.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12194758.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12194758.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12194758.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12194758.xml
-- Model is restarting --
Adjusting last_iter, 1460, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12194758.xml
last_cpu_time: 36730
upload_file_number: 60
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770
The child process has been launched with process id: 3938998
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938998:3938998] [20230125:214245:1674708165:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:43:27 STEP 1440 H=1440:00 +CPU= 37.974
  21:43:49 STEP 1441 H=1441:00 +CPU= 21.778
  21:44:10 STEP 1442 H=1442:00 +CPU= 21.055
  21:44:41 STEP 1443 H=1443:00 +CPU= 31.354
  21:45:03 STEP 1444 H=1444:00 +CPU= 21.616
  21:45:26 STEP 1445 H=1445:00 +CPU= 22.901
  21:46:01 STEP 1446 H=1446:00 +CPU= 34.789
  21:46:23 STEP 1447 H=1447:00 +CPU= 21.412
  21:46:45 STEP 1448 H=1448:00 +CPU= 21.985
  21:47:18 STEP 1449 H=1449:00 +CPU= 32.401
  21:47:40 STEP 1450 H=1450:00 +CPU= 22.741
  21:48:02 STEP 1451 H=1451:00 +CPU= 21.504
  21:48:36 STEP 1452 H=1452:00 +CPU= 33.968
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001452
  21:48:57 STEP 1453 H=1453:00 +CPU= 20.835
  21:49:20 STEP 1454 H=1454:00 +CPU= 22.375
  21:49:53 STEP 1455 H=1455:00 +CPU= 32.728
  21:50:13 STEP 1456 H=1456:00 +CPU= 20.164
  21:50:33 STEP 1457 H=1457:00 +CPU= 19.530
  21:51:06 STEP 1458 H=1458:00 +CPU= 33.110
  21:51:27 STEP 1459 H=1459:00 +CPU= 20.671
  21:51:48 STEP 1460 H=1460:00 +CPU= 20.653
  21:52:20 STEP 1461 H=1461:00 +CPU= 31.564
  21:52:41 STEP 1462 H=1462:00 +CPU= 21.635
  21:53:03 STEP 1463 H=1463:00 +CPU= 21.390
  21:53:41 STEP 1464 H=1464:00 +CPU= 37.363
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:54:03 STEP 1465 H=1465:00 +CPU= 22.245
  21:54:23 STEP 1466 H=1466:00 +CPU= 19.964
  21:54:55 STEP 1467 H=1467:00 +CPU= 30.984
  21:55:18 STEP 1468 H=1468:00 +CPU= 23.543
  21:55:39 STEP 1469 H=1469:00 +CPU= 20.121
  21:56:11 STEP 1470 H=1470:00 +CPU= 31.811
  21:56:35 STEP 1471 H=1471:00 +CPU= 23.477
  21:56:58 STEP 1472 H=1472:00 +CPU= 23.399
  21:57:30 STEP 1473 H=1473:00 +CPU= 31.363
  21:57:51 STEP 1474 H=1474:00 +CPU= 21.046
  21:58:12 STEP 1475 H=1475:00 +CPU= 21.083
  21:58:49 STEP 1476 H=1476:00 +CPU= 36.523
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001476
  21:59:12 STEP 1477 H=1477:00 +CPU= 22.330
  21:59:34 STEP 1478 H=1478:00 +CPU= 22.744
  22:00:08 STEP 1479 H=1479:00 +CPU= 32.927
  22:00:29 STEP 1480 H=1480:00 +CPU= 21.120
  22:00:51 STEP 1481 H=1481:00 +CPU= 21.850
  22:01:24 STEP 1482 H=1482:00 +CPU= 32.732
  22:01:45 STEP 1483 H=1483:00 +CPU= 20.793
  22:02:07 STEP 1484 H=1484:00 +CPU= 21.657
  22:02:41 STEP 1485 H=1485:00 +CPU= 34.326
  22:03:05 STEP 1486 H=1486:00 +CPU= 23.272
  22:03:25 STEP 1487 H=1487:00 +CPU= 20.342
  22:04:01 STEP 1488 H=1488:00 +CPU= 34.808
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_61.zip
Uploading the intermediate file: upload_file_61.zip
  22:04:24 STEP 1489 H=1489:00 +CPU= 23.088
  22:04:46 STEP 1490 H=1490:00 +CPU= 21.509
  22:05:17 STEP 1491 H=1491:00 +CPU= 31.468
  22:05:40 STEP 1492 H=1492:00 +CPU= 22.336
  22:06:02 STEP 1493 H=1493:00 +CPU= 21.370
  22:06:34 STEP 1494 H=1494:00 +CPU= 31.801
  22:06:55 STEP 1495 H=1495:00 +CPU= 20.990
  22:07:18 STEP 1496 H=1496:00 +CPU= 23.086
  22:07:52 STEP 1497 H=1497:00 +CPU= 34.105
  22:08:15 STEP 1498 H=1498:00 +CPU= 22.090
  22:08:35 STEP 1499 H=1499:00 +CPU= 20.041
  22:09:10 STEP 1500 H=1500:00 +CPU= 35.117
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001500
  22:09:34 STEP 1501 H=1501:00 +CPU= 23.097
  22:09:56 STEP 1502 H=1502:00 +CPU= 21.632
  22:10:27 STEP 1503 H=1503:00 +CPU= 31.101
  22:10:49 STEP 1504 H=1504:00 +CPU= 22.186
  22:11:11 STEP 1505 H=1505:00 +CPU= 22.016
  22:11:46 STEP 1506 H=1506:00 +CPU= 34.198
  22:12:07 STEP 1507 H=1507:00 +CPU= 20.441
  22:12:28 STEP 1508 H=1508:00 +CPU= 21.268
  22:13:01 STEP 1509 H=1509:00 +CPU= 33.256
  22:13:24 STEP 1510 H=1510:00 +CPU= 22.212
  22:13:45 STEP 1511 H=1511:00 +CPU= 21.054
  22:14:23 STEP 1512 H=1512:00 +CPU= 37.522
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_62.zip
Uploading the intermediate file: upload_file_62.zip
  22:14:45 STEP 1513 H=1513:00 +CPU= 21.535
  22:15:05 STEP 1514 H=1514:00 +CPU= 20.035
  22:15:38 STEP 1515 H=1515:00 +CPU= 32.538
  22:16:02 STEP 1516 H=1516:00 +CPU= 23.456
  22:16:21 STEP 1517 H=1517:00 +CPU= 19.509
  22:16:52 STEP 1518 H=1518:00 +CPU= 30.008
  22:17:15 STEP 1519 H=1519:00 +CPU= 23.458
  22:17:39 STEP 1520 H=1520:00 +CPU= 23.190
  22:18:09 STEP 1521 H=1521:00 +CPU= 30.330
  22:18:31 STEP 1522 H=1522:00 +CPU= 21.783
  22:18:55 STEP 1523 H=1523:00 +CPU= 23.942
  22:19:31 STEP 1524 H=1524:00 +CPU= 35.057
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001524
  22:19:53 STEP 1525 H=1525:00 +CPU= 21.465
  22:20:17 STEP 1526 H=1526:00 +CPU= 24.030
  22:20:52 STEP 1527 H=1527:00 +CPU= 34.861
  22:21:11 STEP 1528 H=1528:00 +CPU= 19.358
  22:21:33 STEP 1529 H=1529:00 +CPU= 20.962
  22:22:10 STEP 1530 H=1530:00 +CPU= 36.752
  22:22:29 STEP 1531 H=1531:00 +CPU= 19.385
  22:22:51 STEP 1532 H=1532:00 +CPU= 21.461
  22:23:25 STEP 1533 H=1533:00 +CPU= 33.712
  22:23:46 STEP 1534 H=1534:00 +CPU= 20.788
  22:24:07 STEP 1535 H=1535:00 +CPU= 21.393
  22:24:45 STEP 1536 H=1536:00 +CPU= 37.177
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_63.zip
Uploading the intermediate file: upload_file_63.zip
  22:25:06 STEP 1537 H=1537:00 +CPU= 21.063
  22:25:27 STEP 1538 H=1538:00 +CPU= 21.293
  22:26:02 STEP 1539 H=1539:00 +CPU= 33.810
  22:26:25 STEP 1540 H=1540:00 +CPU= 23.252
  22:26:45 STEP 1541 H=1541:00 +CPU= 19.824
  22:27:18 STEP 1542 H=1542:00 +CPU= 32.181
  22:27:41 STEP 1543 H=1543:00 +CPU= 22.867
  22:28:03 STEP 1544 H=1544:00 +CPU= 21.810
  22:28:34 STEP 1545 H=1545:00 +CPU= 31.500
  22:28:55 STEP 1546 H=1546:00 +CPU= 20.870
  22:29:19 STEP 1547 H=1547:00 +CPU= 23.189
  22:29:57 STEP 1548 H=1548:00 +CPU= 37.154
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001548
  22:30:17 STEP 1549 H=1549:00 +CPU= 20.098
  22:30:36 STEP 1550 H=1550:00 +CPU= 18.833
  22:31:06 STEP 1551 H=1551:00 +CPU= 30.511
  22:31:28 STEP 1552 H=1552:00 +CPU= 21.074
  22:31:49 STEP 1553 H=1553:00 +CPU= 21.485
  22:32:21 STEP 1554 H=1554:00 +CPU= 31.834
  22:32:42 STEP 1555 H=1555:00 +CPU= 20.278
  22:33:06 STEP 1556 H=1556:00 +CPU= 23.807
  22:33:40 STEP 1557 H=1557:00 +CPU= 33.447
  22:34:00 STEP 1558 H=1558:00 +CPU= 19.974
  22:34:22 STEP 1559 H=1559:00 +CPU= 22.613
  22:35:01 STEP 1560 H=1560:00 +CPU= 38.420
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_64.zip
Uploading the intermediate file: upload_file_64.zip
  22:35:22 STEP 1561 H=1561:00 +CPU= 20.467
  22:35:43 STEP 1562 H=1562:00 +CPU= 21.533
  22:36:18 STEP 1563 H=1563:00 +CPU= 33.481
  22:36:38 STEP 1564 H=1564:00 +CPU= 20.589
  22:37:01 STEP 1565 H=1565:00 +CPU= 21.806
  22:37:36 STEP 1566 H=1566:00 +CPU= 35.257
  22:37:58 STEP 1567 H=1567:00 +CPU= 21.444
  22:38:20 STEP 1568 H=1568:00 +CPU= 22.117
  22:38:55 STEP 1569 H=1569:00 +CPU= 34.292
  22:39:16 STEP 1570 H=1570:00 +CPU= 20.699
  22:39:36 STEP 1571 H=1571:00 +CPU= 20.156
  22:40:14 STEP 1572 H=1572:00 +CPU= 37.615
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001572
  22:40:35 STEP 1573 H=1573:00 +CPU= 20.805
  22:40:56 STEP 1574 H=1574:00 +CPU= 21.123
  22:41:29 STEP 1575 H=1575:00 +CPU= 32.229
  22:41:52 STEP 1576 H=1576:00 +CPU= 23.395
  22:42:14 STEP 1577 H=1577:00 +CPU= 21.519
  22:42:47 STEP 1578 H=1578:00 +CPU= 32.614
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2009050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0114
(argv4) batchid: 978
(argv5) wuid: 12194758
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/8
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_44ae5f80e9496dd8778b9bea20bea134 to: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0114_2009050100_123_978_12194758.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/8/oifs_43r3_ps_0114_2009050100_123_978_12194758.zip
ic_ancil_file: ic_ancil_12194758
ifsdata_file: ifsdata_12194758
climate_data_file: clim_data_12194758
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_caff69d5e7c798bae75b5ac4a0228f16 to: /var/lib/boinc-client/slots/8/ic_ancil_12194758.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/8/ic_ancil_12194758.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f378b9847cd8695263ee8b3e30f00712 to: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12194758.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/8/ifsdata/ifsdata_12194758.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/8/159l_2/clim_data_12194758.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/8/159l_2/clim_data_12194758.zip
Checking for progress XML file: /var/lib/boinc-client/slots/8/progress_file_12194758.xml
Opened progress file ok : /var/lib/boinc-client/slots/8/progress_file_12194758.xml
-- Model is restarting --
Adjusting last_iter, 1579, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/8/progress_file_12194758.xml
last_cpu_time: 40308
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770
The child process has been launched with process id: 3949521
Executing the command: /var/lib/boinc-client/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949521:3949521] [20230125:224406:1674711846:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:44:48 STEP 1560 H=1560:00 +CPU= 37.771
  22:45:08 STEP 1561 H=1561:00 +CPU= 20.208
  22:45:27 STEP 1562 H=1562:00 +CPU= 18.957
  22:45:59 STEP 1563 H=1563:00 +CPU= 31.989
  22:46:21 STEP 1564 H=1564:00 +CPU= 21.553
  22:46:41 STEP 1565 H=1565:00 +CPU= 20.165
  22:47:13 STEP 1566 H=1566:00 +CPU= 31.659
  22:47:36 STEP 1567 H=1567:00 +CPU= 22.729
  22:47:57 STEP 1568 H=1568:00 +CPU= 20.906
  22:48:28 STEP 1569 H=1569:00 +CPU= 30.776
  22:48:52 STEP 1570 H=1570:00 +CPU= 23.731
  22:49:14 STEP 1571 H=1571:00 +CPU= 21.416
  22:49:47 STEP 1572 H=1572:00 +CPU= 33.400
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001572
  22:50:08 STEP 1573 H=1573:00 +CPU= 21.021
  22:50:31 STEP 1574 H=1574:00 +CPU= 22.176
  22:51:02 STEP 1575 H=1575:00 +CPU= 30.795
  22:51:22 STEP 1576 H=1576:00 +CPU= 19.848
  22:51:43 STEP 1577 H=1577:00 +CPU= 21.333
  22:52:17 STEP 1578 H=1578:00 +CPU= 33.409
  22:52:36 STEP 1579 H=1579:00 +CPU= 18.725
  22:52:56 STEP 1580 H=1580:00 +CPU= 19.900
  22:53:30 STEP 1581 H=1581:00 +CPU= 33.663
  22:53:51 STEP 1582 H=1582:00 +CPU= 21.313
  22:54:12 STEP 1583 H=1583:00 +CPU= 20.952
  22:54:48 STEP 1584 H=1584:00 +CPU= 35.451
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/8/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12194758/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0114_2009050100_123_978_12194758_1_r1118153770_65.zip
Uploading the intermediate file: upload_file_65.zip
  22:55:09 STEP 1585 H=1585:00 +CPU= 20.794
  22:55:28 STEP 1586 H=1586:00 +CPU= 19.709
  22:55:57 STEP 1587 H=1587:00 +CPU= 28.231
  22:56:15 STEP 1588 H=1588:00 +CPU= 18.178
  22:56:33 STEP 1589 H=1589:00 +CPU= 17.939
  22:57:04 STEP 1590 H=1590:00 +CPU= 30.700
  22:57:24 STEP 1591 H=1591:00 +CPU= 19.512
  22:57:44 STEP 1592 H=1592:00 +CPU= 19.858
  22:58:13 STEP 1593 H=1593:00 +CPU= 29.194
  22:58:33 STEP 1594 H=1594:00 +CPU= 19.257
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:41:14 1532237 22303610 oifs_43r3_ps_0114_2009050100_123_978_12194758_1 1,728,000 10,824 0.0063
09 May 2023 16:41:11 1532237 22303610 oifs_43r3_ps_0114_2009050100_123_978_12194758_1 864,000 5,421 0.0063


©2024 climateprediction.net