climateprediction.net home page
Task 22303609

Task 22303609

Name oifs_43r3_ps_0108_2004050100_123_973_12189752_1
Workunit 12189752
Created 25 Jan 2023, 18:10:28 UTC
Sent 25 Jan 2023, 18:12:37 UTC
Report deadline 26 Mar 2023, 18:12:37 UTC
Received 26 Jan 2023, 7:48:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532237
Run time 11 hours 31 min 57 sec
CPU time 11 hours 20 min 37 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,407.29 MB
Peak swap size 4,978.14 MB
Peak disk usage 7,688.48 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ots/9/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001260
  20:14:11 STEP 1261 H=1261:00 +CPU= 22.571
  20:14:34 STEP 1262 H=1262:00 +CPU= 22.529
  20:15:06 STEP 1263 H=1263:00 +CPU= 31.648
  20:15:27 STEP 1264 H=1264:00 +CPU= 21.355
  20:15:48 STEP 1265 H=1265:00 +CPU= 20.080
  20:16:19 STEP 1266 H=1266:00 +CPU= 30.830
  20:16:42 STEP 1267 H=1267:00 +CPU= 22.361
  20:17:05 STEP 1268 H=1268:00 +CPU= 23.253
  20:17:38 STEP 1269 H=1269:00 +CPU= 32.738
  20:18:00 STEP 1270 H=1270:00 +CPU= 21.905
  20:18:22 STEP 1271 H=1271:00 +CPU= 21.396
  20:19:01 STEP 1272 H=1272:00 +CPU= 38.835
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_52.zip
Uploading the intermediate file: upload_file_52.zip
  20:19:23 STEP 1273 H=1273:00 +CPU= 21.494
  20:19:44 STEP 1274 H=1274:00 +CPU= 21.561
  20:20:18 STEP 1275 H=1275:00 +CPU= 33.593
  20:20:42 STEP 1276 H=1276:00 +CPU= 22.984
  20:21:02 STEP 1277 H=1277:00 +CPU= 20.807
  20:21:34 STEP 1278 H=1278:00 +CPU= 31.429
  20:21:56 STEP 1279 H=1279:00 +CPU= 21.601
  20:22:20 STEP 1280 H=1280:00 +CPU= 23.491
  20:22:52 STEP 1281 H=1281:00 +CPU= 31.827
  20:23:12 STEP 1282 H=1282:00 +CPU= 20.048
  20:23:35 STEP 1283 H=1283:00 +CPU= 22.953
  20:24:12 STEP 1284 H=1284:00 +CPU= 36.474
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001284
  20:24:33 STEP 1285 H=1285:00 +CPU= 21.072
  20:24:56 STEP 1286 H=1286:00 +CPU= 23.197
  20:25:29 STEP 1287 H=1287:00 +CPU= 31.732
  20:25:50 STEP 1288 H=1288:00 +CPU= 21.358
  20:26:13 STEP 1289 H=1289:00 +CPU= 22.800
  20:26:46 STEP 1290 H=1290:00 +CPU= 32.539
  20:27:06 STEP 1291 H=1291:00 +CPU= 19.625
  20:27:29 STEP 1292 H=1292:00 +CPU= 23.091
  20:28:01 STEP 1293 H=1293:00 +CPU= 31.585
  20:28:22 STEP 1294 H=1294:00 +CPU= 20.797
  20:28:45 STEP 1295 H=1295:00 +CPU= 22.331
  20:29:21 STEP 1296 H=1296:00 +CPU= 35.740
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_53.zip
Uploading the intermediate file: upload_file_53.zip
  20:29:41 STEP 1297 H=1297:00 +CPU= 20.056
  20:30:03 STEP 1298 H=1298:00 +CPU= 22.087
  20:30:38 STEP 1299 H=1299:00 +CPU= 34.860
  20:31:00 STEP 1300 H=1300:00 +CPU= 21.734
  20:31:22 STEP 1301 H=1301:00 +CPU= 21.865
  20:31:55 STEP 1302 H=1302:00 +CPU= 32.181
  20:32:16 STEP 1303 H=1303:00 +CPU= 21.134
  20:32:39 STEP 1304 H=1304:00 +CPU= 22.404
  20:33:11 STEP 1305 H=1305:00 +CPU= 32.030
  20:33:34 STEP 1306 H=1306:00 +CPU= 22.937
  20:33:56 STEP 1307 H=1307:00 +CPU= 21.617
  20:34:34 STEP 1308 H=1308:00 +CPU= 37.482
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001308
  20:34:56 STEP 1309 H=1309:00 +CPU= 21.784
  20:35:17 STEP 1310 H=1310:00 +CPU= 20.976
  20:35:49 STEP 1311 H=1311:00 +CPU= 32.313
  20:36:13 STEP 1312 H=1312:00 +CPU= 23.149
  20:36:35 STEP 1313 H=1313:00 +CPU= 22.318
  20:37:08 STEP 1314 H=1314:00 +CPU= 32.537
  20:37:30 STEP 1315 H=1315:00 +CPU= 21.999
  20:37:54 STEP 1316 H=1316:00 +CPU= 23.692
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0108
(argv4) batchid: 973
(argv5) wuid: 12189752
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_425c8e492094c790dfef5ec6b5a94837 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0108_2004050100_123_973_12189752.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0108_2004050100_123_973_12189752.zip
ic_ancil_file: ic_ancil_12189752
ifsdata_file: ifsdata_12189752
climate_data_file: clim_data_12189752
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2df444aa159f3f5515a2a0a3353a14f0 to: /var/lib/boinc-client/slots/9/ic_ancil_12189752.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12189752.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ff2b28f206bc257068084d7d67ea90b3 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12189752.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12189752.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12189752.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12189752.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12189752.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12189752.xml
-- Model is restarting --
Adjusting last_iter, 1316, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12189752.xml
last_cpu_time: 32971
upload_file_number: 54
last_iter: 1295
last_upload: 4665600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108
The child process has been launched with process id: 3928548
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3928548:3928548] [20230125:203900:1674704340:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:39:44 STEP 1296 H=1296:00 +CPU= 39.495
  20:40:05 STEP 1297 H=1297:00 +CPU= 21.308
  20:40:26 STEP 1298 H=1298:00 +CPU= 21.081
  20:41:00 STEP 1299 H=1299:00 +CPU= 33.515
  20:41:22 STEP 1300 H=1300:00 +CPU= 21.630
  20:41:43 STEP 1301 H=1301:00 +CPU= 21.029
  20:42:17 STEP 1302 H=1302:00 +CPU= 33.430
  20:42:37 STEP 1303 H=1303:00 +CPU= 20.270
  20:42:59 STEP 1304 H=1304:00 +CPU= 22.078
  20:43:33 STEP 1305 H=1305:00 +CPU= 33.517
  20:43:55 STEP 1306 H=1306:00 +CPU= 22.326
  20:44:17 STEP 1307 H=1307:00 +CPU= 21.193
  20:44:53 STEP 1308 H=1308:00 +CPU= 35.718
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001308
  20:45:15 STEP 1309 H=1309:00 +CPU= 22.165
  20:45:37 STEP 1310 H=1310:00 +CPU= 21.862
  20:46:12 STEP 1311 H=1311:00 +CPU= 34.437
  20:46:34 STEP 1312 H=1312:00 +CPU= 21.860
  20:46:54 STEP 1313 H=1313:00 +CPU= 20.324
  20:47:27 STEP 1314 H=1314:00 +CPU= 32.739
  20:47:48 STEP 1315 H=1315:00 +CPU= 20.106
  20:48:08 STEP 1316 H=1316:00 +CPU= 20.499
  20:48:44 STEP 1317 H=1317:00 +CPU= 35.437
  20:49:07 STEP 1318 H=1318:00 +CPU= 22.812
  20:49:29 STEP 1319 H=1319:00 +CPU= 21.589
  20:50:07 STEP 1320 H=1320:00 +CPU= 37.746
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_54.zip
Uploading the intermediate file: upload_file_54.zip
  20:50:30 STEP 1321 H=1321:00 +CPU= 22.656
  20:50:50 STEP 1322 H=1322:00 +CPU= 19.851
  20:51:21 STEP 1323 H=1323:00 +CPU= 30.937
  20:51:43 STEP 1324 H=1324:00 +CPU= 22.275
  20:52:03 STEP 1325 H=1325:00 +CPU= 19.879
  20:52:37 STEP 1326 H=1326:00 +CPU= 33.639
  20:53:00 STEP 1327 H=1327:00 +CPU= 22.536
  20:53:24 STEP 1328 H=1328:00 +CPU= 23.989
  20:53:57 STEP 1329 H=1329:00 +CPU= 33.012
  20:54:19 STEP 1330 H=1330:00 +CPU= 21.623
  20:54:40 STEP 1331 H=1331:00 +CPU= 20.509
  20:55:17 STEP 1332 H=1332:00 +CPU= 36.996
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001332
  20:55:37 STEP 1333 H=1333:00 +CPU= 20.295
  20:55:57 STEP 1334 H=1334:00 +CPU= 19.104
  20:56:30 STEP 1335 H=1335:00 +CPU= 33.272
  20:56:54 STEP 1336 H=1336:00 +CPU= 23.004
  20:57:13 STEP 1337 H=1337:00 +CPU= 19.796
  20:57:47 STEP 1338 H=1338:00 +CPU= 33.282
  20:58:09 STEP 1339 H=1339:00 +CPU= 21.581
  20:58:30 STEP 1340 H=1340:00 +CPU= 20.469
  20:59:01 STEP 1341 H=1341:00 +CPU= 30.687
  20:59:24 STEP 1342 H=1342:00 +CPU= 23.495
  20:59:47 STEP 1343 H=1343:00 +CPU= 22.672
  21:00:24 STEP 1344 H=1344:00 +CPU= 36.271
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_55.zip
Uploading the intermediate file: upload_file_55.zip
  21:00:47 STEP 1345 H=1345:00 +CPU= 23.236
  21:01:08 STEP 1346 H=1346:00 +CPU= 20.752
  21:01:39 STEP 1347 H=1347:00 +CPU= 30.191
  21:02:01 STEP 1348 H=1348:00 +CPU= 21.884
  21:02:24 STEP 1349 H=1349:00 +CPU= 22.814
  21:02:56 STEP 1350 H=1350:00 +CPU= 32.333
  21:03:17 STEP 1351 H=1351:00 +CPU= 20.115
  21:03:40 STEP 1352 H=1352:00 +CPU= 22.965
  21:04:12 STEP 1353 H=1353:00 +CPU= 31.656
  21:04:33 STEP 1354 H=1354:00 +CPU= 21.273
  21:04:56 STEP 1355 H=1355:00 +CPU= 22.186
  21:05:32 STEP 1356 H=1356:00 +CPU= 36.314
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001356
  21:05:53 STEP 1357 H=1357:00 +CPU= 20.568
  21:06:15 STEP 1358 H=1358:00 +CPU= 21.543
  21:06:48 STEP 1359 H=1359:00 +CPU= 33.003
  21:07:10 STEP 1360 H=1360:00 +CPU= 21.392
  21:07:30 STEP 1361 H=1361:00 +CPU= 20.372
  21:08:06 STEP 1362 H=1362:00 +CPU= 34.838
  21:08:26 STEP 1363 H=1363:00 +CPU= 19.866
  21:08:45 STEP 1364 H=1364:00 +CPU= 19.555
  21:09:18 STEP 1365 H=1365:00 +CPU= 31.953
  21:09:37 STEP 1366 H=1366:00 +CPU= 19.517
  21:09:59 STEP 1367 H=1367:00 +CPU= 21.651
  21:10:36 STEP 1368 H=1368:00 +CPU= 36.313
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_56.zip
Uploading the intermediate file: upload_file_56.zip
  21:10:56 STEP 1369 H=1369:00 +CPU= 20.263
  21:11:17 STEP 1370 H=1370:00 +CPU= 21.177
  21:11:50 STEP 1371 H=1371:00 +CPU= 32.897
  21:12:13 STEP 1372 H=1372:00 +CPU= 22.016
  21:12:36 STEP 1373 H=1373:00 +CPU= 22.916
  21:13:10 STEP 1374 H=1374:00 +CPU= 34.326
  21:13:30 STEP 1375 H=1375:00 +CPU= 19.873
  21:13:53 STEP 1376 H=1376:00 +CPU= 22.559
  21:14:24 STEP 1377 H=1377:00 +CPU= 30.593
  21:14:45 STEP 1378 H=1378:00 +CPU= 21.121
  21:15:08 STEP 1379 H=1379:00 +CPU= 22.136
  21:15:44 STEP 1380 H=1380:00 +CPU= 36.153
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001380
  21:16:06 STEP 1381 H=1381:00 +CPU= 21.277
  21:16:28 STEP 1382 H=1382:00 +CPU= 21.880
  21:17:00 STEP 1383 H=1383:00 +CPU= 32.006
  21:17:23 STEP 1384 H=1384:00 +CPU= 23.157
  21:17:45 STEP 1385 H=1385:00 +CPU= 21.433
  21:18:18 STEP 1386 H=1386:00 +CPU= 32.974
  21:18:39 STEP 1387 H=1387:00 +CPU= 20.342
  21:19:02 STEP 1388 H=1388:00 +CPU= 23.029
  21:19:34 STEP 1389 H=1389:00 +CPU= 31.701
  21:19:53 STEP 1390 H=1390:00 +CPU= 19.046
  21:20:17 STEP 1391 H=1391:00 +CPU= 22.881
  21:20:57 STEP 1392 H=1392:00 +CPU= 39.587
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_57.zip
Uploading the intermediate file: upload_file_57.zip
  21:21:17 STEP 1393 H=1393:00 +CPU= 20.301
  21:21:39 STEP 1394 H=1394:00 +CPU= 21.381
  21:22:12 STEP 1395 H=1395:00 +CPU= 33.231
  21:22:34 STEP 1396 H=1396:00 +CPU= 21.988
  21:22:54 STEP 1397 H=1397:00 +CPU= 19.376
  21:23:24 STEP 1398 H=1398:00 +CPU= 29.615
  21:23:45 STEP 1399 H=1399:00 +CPU= 21.328
  21:24:09 STEP 1400 H=1400:00 +CPU= 23.142
  21:24:40 STEP 1401 H=1401:00 +CPU= 31.565
  21:25:01 STEP 1402 H=1402:00 +CPU= 20.952
  21:25:25 STEP 1403 H=1403:00 +CPU= 23.287
  21:26:01 STEP 1404 H=1404:00 +CPU= 36.078
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001404
  21:26:23 STEP 1405 H=1405:00 +CPU= 20.995
  21:26:45 STEP 1406 H=1406:00 +CPU= 22.680
  21:27:21 STEP 1407 H=1407:00 +CPU= 34.777
  21:27:42 STEP 1408 H=1408:00 +CPU= 21.502
  21:28:03 STEP 1409 H=1409:00 +CPU= 21.032
  21:28:38 STEP 1410 H=1410:00 +CPU= 34.516
  21:29:01 STEP 1411 H=1411:00 +CPU= 22.771
  21:29:22 STEP 1412 H=1412:00 +CPU= 20.334
  21:29:55 STEP 1413 H=1413:00 +CPU= 32.912
  21:30:18 STEP 1414 H=1414:00 +CPU= 23.119
  21:30:40 STEP 1415 H=1415:00 +CPU= 21.894
  21:31:17 STEP 1416 H=1416:00 +CPU= 36.557
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_58.zip
Uploading the intermediate file: upload_file_58.zip
  21:31:40 STEP 1417 H=1417:00 +CPU= 22.449
  21:32:03 STEP 1418 H=1418:00 +CPU= 22.815
  21:32:34 STEP 1419 H=1419:00 +CPU= 31.029
  21:32:56 STEP 1420 H=1420:00 +CPU= 21.375
  21:33:19 STEP 1421 H=1421:00 +CPU= 23.306
  21:33:54 STEP 1422 H=1422:00 +CPU= 33.797
  21:34:14 STEP 1423 H=1423:00 +CPU= 20.745
  21:34:39 STEP 1424 H=1424:00 +CPU= 23.869
  21:35:13 STEP 1425 H=1425:00 +CPU= 33.553
  21:35:33 STEP 1426 H=1426:00 +CPU= 19.654
  21:35:55 STEP 1427 H=1427:00 +CPU= 21.784
  21:36:32 STEP 1428 H=1428:00 +CPU= 36.640
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001428
  21:36:51 STEP 1429 H=1429:00 +CPU= 19.197
  21:37:11 STEP 1430 H=1430:00 +CPU= 20.390
  21:37:48 STEP 1431 H=1431:00 +CPU= 36.003
  21:38:10 STEP 1432 H=1432:00 +CPU= 22.143
  21:38:31 STEP 1433 H=1433:00 +CPU= 20.279
  21:39:05 STEP 1434 H=1434:00 +CPU= 33.586
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0108
(argv4) batchid: 973
(argv5) wuid: 12189752
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_425c8e492094c790dfef5ec6b5a94837 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0108_2004050100_123_973_12189752.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0108_2004050100_123_973_12189752.zip
ic_ancil_file: ic_ancil_12189752
ifsdata_file: ifsdata_12189752
climate_data_file: clim_data_12189752
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2df444aa159f3f5515a2a0a3353a14f0 to: /var/lib/boinc-client/slots/9/ic_ancil_12189752.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12189752.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ff2b28f206bc257068084d7d67ea90b3 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12189752.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12189752.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12189752.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12189752.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12189752.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12189752.xml
-- Model is restarting --
Adjusting last_iter, 1435, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12189752.xml
last_cpu_time: 36548
upload_file_number: 59
last_iter: 1415
last_upload: 5097600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108
The child process has been launched with process id: 3938965
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3938965:3938965] [20230125:214021:1674708021:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:40:59 STEP 1416 H=1416:00 +CPU= 34.812
  21:41:23 STEP 1417 H=1417:00 +CPU= 23.095
  21:41:46 STEP 1418 H=1418:00 +CPU= 22.724
  21:42:19 STEP 1419 H=1419:00 +CPU= 32.818
  21:42:40 STEP 1420 H=1420:00 +CPU= 21.536
  21:43:03 STEP 1421 H=1421:00 +CPU= 22.573
  21:43:37 STEP 1422 H=1422:00 +CPU= 33.584
  21:43:59 STEP 1423 H=1423:00 +CPU= 22.126
  21:44:21 STEP 1424 H=1424:00 +CPU= 21.680
  21:44:52 STEP 1425 H=1425:00 +CPU= 31.394
  21:45:16 STEP 1426 H=1426:00 +CPU= 23.717
  21:45:40 STEP 1427 H=1427:00 +CPU= 23.035
  21:46:16 STEP 1428 H=1428:00 +CPU= 36.455
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001428
  21:46:37 STEP 1429 H=1429:00 +CPU= 20.400
  21:46:59 STEP 1430 H=1430:00 +CPU= 22.130
  21:47:32 STEP 1431 H=1431:00 +CPU= 33.266
  21:47:55 STEP 1432 H=1432:00 +CPU= 22.260
  21:48:16 STEP 1433 H=1433:00 +CPU= 21.195
  21:48:49 STEP 1434 H=1434:00 +CPU= 32.724
  21:49:12 STEP 1435 H=1435:00 +CPU= 22.249
  21:49:34 STEP 1436 H=1436:00 +CPU= 22.151
  21:50:06 STEP 1437 H=1437:00 +CPU= 31.765
  21:50:26 STEP 1438 H=1438:00 +CPU= 19.636
  21:50:47 STEP 1439 H=1439:00 +CPU= 21.255
  21:51:25 STEP 1440 H=1440:00 +CPU= 37.552
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_59.zip
Uploading the intermediate file: upload_file_59.zip
  21:51:46 STEP 1441 H=1441:00 +CPU= 21.217
  21:52:07 STEP 1442 H=1442:00 +CPU= 20.795
  21:52:40 STEP 1443 H=1443:00 +CPU= 32.348
  21:53:02 STEP 1444 H=1444:00 +CPU= 21.667
  21:53:22 STEP 1445 H=1445:00 +CPU= 20.294
  21:53:57 STEP 1446 H=1446:00 +CPU= 34.288
  21:54:18 STEP 1447 H=1447:00 +CPU= 21.347
  21:54:39 STEP 1448 H=1448:00 +CPU= 20.774
  21:55:12 STEP 1449 H=1449:00 +CPU= 32.259
  21:55:35 STEP 1450 H=1450:00 +CPU= 23.026
  21:55:56 STEP 1451 H=1451:00 +CPU= 20.592
  21:56:32 STEP 1452 H=1452:00 +CPU= 35.632
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001452
  21:56:55 STEP 1453 H=1453:00 +CPU= 23.043
  21:57:17 STEP 1454 H=1454:00 +CPU= 22.212
  21:57:49 STEP 1455 H=1455:00 +CPU= 31.278
  21:58:11 STEP 1456 H=1456:00 +CPU= 21.367
  21:58:33 STEP 1457 H=1457:00 +CPU= 22.176
  21:59:08 STEP 1458 H=1458:00 +CPU= 34.492
  21:59:32 STEP 1459 H=1459:00 +CPU= 23.760
  21:59:53 STEP 1460 H=1460:00 +CPU= 21.464
  22:00:27 STEP 1461 H=1461:00 +CPU= 33.010
  22:00:49 STEP 1462 H=1462:00 +CPU= 21.722
  22:01:10 STEP 1463 H=1463:00 +CPU= 21.254
  22:01:49 STEP 1464 H=1464:00 +CPU= 38.410
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_60.zip
Uploading the intermediate file: upload_file_60.zip
  22:02:10 STEP 1465 H=1465:00 +CPU= 21.573
  22:02:33 STEP 1466 H=1466:00 +CPU= 22.288
  22:03:06 STEP 1467 H=1467:00 +CPU= 33.000
  22:03:27 STEP 1468 H=1468:00 +CPU= 20.317
  22:03:45 STEP 1469 H=1469:00 +CPU= 18.293
  22:04:18 STEP 1470 H=1470:00 +CPU= 32.656
  22:04:41 STEP 1471 H=1471:00 +CPU= 22.416
  22:05:03 STEP 1472 H=1472:00 +CPU= 21.850
  22:05:35 STEP 1473 H=1473:00 +CPU= 32.341
  22:06:00 STEP 1474 H=1474:00 +CPU= 23.766
  22:06:20 STEP 1475 H=1475:00 +CPU= 20.170
  22:06:55 STEP 1476 H=1476:00 +CPU= 35.031
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001476
  22:07:17 STEP 1477 H=1477:00 +CPU= 22.027
  22:07:41 STEP 1478 H=1478:00 +CPU= 23.252
  22:08:14 STEP 1479 H=1479:00 +CPU= 32.580
  22:08:34 STEP 1480 H=1480:00 +CPU= 20.586
  22:08:54 STEP 1481 H=1481:00 +CPU= 19.868
  22:09:30 STEP 1482 H=1482:00 +CPU= 34.598
  22:09:52 STEP 1483 H=1483:00 +CPU= 21.858
  22:10:13 STEP 1484 H=1484:00 +CPU= 21.471
  22:10:46 STEP 1485 H=1485:00 +CPU= 32.419
  22:11:09 STEP 1486 H=1486:00 +CPU= 22.570
  22:11:32 STEP 1487 H=1487:00 +CPU= 23.358
  22:12:09 STEP 1488 H=1488:00 +CPU= 36.953
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_61.zip
Uploading the intermediate file: upload_file_61.zip
  22:12:30 STEP 1489 H=1489:00 +CPU= 20.201
  22:12:51 STEP 1490 H=1490:00 +CPU= 21.012
  22:13:25 STEP 1491 H=1491:00 +CPU= 34.137
  22:13:47 STEP 1492 H=1492:00 +CPU= 21.619
  22:14:07 STEP 1493 H=1493:00 +CPU= 20.037
  22:14:42 STEP 1494 H=1494:00 +CPU= 34.168
  22:15:02 STEP 1495 H=1495:00 +CPU= 19.959
  22:15:23 STEP 1496 H=1496:00 +CPU= 21.261
  22:15:58 STEP 1497 H=1497:00 +CPU= 33.749
  22:16:19 STEP 1498 H=1498:00 +CPU= 20.990
  22:16:38 STEP 1499 H=1499:00 +CPU= 18.967
  22:17:15 STEP 1500 H=1500:00 +CPU= 36.943
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001500
  22:17:38 STEP 1501 H=1501:00 +CPU= 22.498
  22:17:59 STEP 1502 H=1502:00 +CPU= 20.796
  22:18:31 STEP 1503 H=1503:00 +CPU= 31.579
  22:18:56 STEP 1504 H=1504:00 +CPU= 24.899
  22:19:19 STEP 1505 H=1505:00 +CPU= 22.517
  22:19:51 STEP 1506 H=1506:00 +CPU= 32.611
  22:20:14 STEP 1507 H=1507:00 +CPU= 22.622
  22:20:38 STEP 1508 H=1508:00 +CPU= 23.963
  22:21:10 STEP 1509 H=1509:00 +CPU= 30.812
  22:21:31 STEP 1510 H=1510:00 +CPU= 21.273
  22:21:55 STEP 1511 H=1511:00 +CPU= 23.940
  22:22:34 STEP 1512 H=1512:00 +CPU= 38.431
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_62.zip
Uploading the intermediate file: upload_file_62.zip
  22:22:54 STEP 1513 H=1513:00 +CPU= 20.287
  22:23:15 STEP 1514 H=1514:00 +CPU= 20.909
  22:23:50 STEP 1515 H=1515:00 +CPU= 33.944
  22:24:11 STEP 1516 H=1516:00 +CPU= 20.999
  22:24:30 STEP 1517 H=1517:00 +CPU= 19.068
  22:25:05 STEP 1518 H=1518:00 +CPU= 34.365
  22:25:25 STEP 1519 H=1519:00 +CPU= 20.622
  22:25:46 STEP 1520 H=1520:00 +CPU= 20.768
  22:26:20 STEP 1521 H=1521:00 +CPU= 33.258
  22:26:43 STEP 1522 H=1522:00 +CPU= 22.204
  22:27:03 STEP 1523 H=1523:00 +CPU= 20.646
  22:27:39 STEP 1524 H=1524:00 +CPU= 35.250
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001524
  22:28:01 STEP 1525 H=1525:00 +CPU= 22.219
  22:28:23 STEP 1526 H=1526:00 +CPU= 22.022
  22:28:55 STEP 1527 H=1527:00 +CPU= 31.162
  22:29:19 STEP 1528 H=1528:00 +CPU= 24.079
  22:29:41 STEP 1529 H=1529:00 +CPU= 21.824
  22:30:13 STEP 1530 H=1530:00 +CPU= 32.253
  22:30:32 STEP 1531 H=1531:00 +CPU= 18.636
  22:30:52 STEP 1532 H=1532:00 +CPU= 19.597
  22:31:23 STEP 1533 H=1533:00 +CPU= 30.607
  22:31:46 STEP 1534 H=1534:00 +CPU= 22.989
  22:32:07 STEP 1535 H=1535:00 +CPU= 21.153
  22:32:44 STEP 1536 H=1536:00 +CPU= 35.919
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_63.zip
Uploading the intermediate file: upload_file_63.zip
  22:33:06 STEP 1537 H=1537:00 +CPU= 22.219
  22:33:30 STEP 1538 H=1538:00 +CPU= 23.473
  22:34:00 STEP 1539 H=1539:00 +CPU= 29.795
  22:34:23 STEP 1540 H=1540:00 +CPU= 22.966
  22:34:47 STEP 1541 H=1541:00 +CPU= 23.735
  22:35:18 STEP 1542 H=1542:00 +CPU= 31.375
  22:35:39 STEP 1543 H=1543:00 +CPU= 20.876
  22:36:03 STEP 1544 H=1544:00 +CPU= 23.103
  22:36:36 STEP 1545 H=1545:00 +CPU= 32.658
  22:36:58 STEP 1546 H=1546:00 +CPU= 22.135
  22:37:21 STEP 1547 H=1547:00 +CPU= 22.630
  22:37:59 STEP 1548 H=1548:00 +CPU= 37.275
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001548
  22:38:20 STEP 1549 H=1549:00 +CPU= 21.537
  22:38:43 STEP 1550 H=1550:00 +CPU= 22.358
  22:39:16 STEP 1551 H=1551:00 +CPU= 32.977
  22:39:38 STEP 1552 H=1552:00 +CPU= 21.137
  22:40:00 STEP 1553 H=1553:00 +CPU= 22.351
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2004050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0108
(argv4) batchid: 973
(argv5) wuid: 12189752
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/9
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_425c8e492094c790dfef5ec6b5a94837 to: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0108_2004050100_123_973_12189752.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/9/oifs_43r3_ps_0108_2004050100_123_973_12189752.zip
ic_ancil_file: ic_ancil_12189752
ifsdata_file: ifsdata_12189752
climate_data_file: clim_data_12189752
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2df444aa159f3f5515a2a0a3353a14f0 to: /var/lib/boinc-client/slots/9/ic_ancil_12189752.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/9/ic_ancil_12189752.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_ff2b28f206bc257068084d7d67ea90b3 to: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12189752.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/9/ifsdata/ifsdata_12189752.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/9/159l_2/clim_data_12189752.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/9/159l_2/clim_data_12189752.zip
Checking for progress XML file: /var/lib/boinc-client/slots/9/progress_file_12189752.xml
Opened progress file ok : /var/lib/boinc-client/slots/9/progress_file_12189752.xml
-- Model is restarting --
Adjusting last_iter, 1554, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/9/progress_file_12189752.xml
last_cpu_time: 40117
upload_file_number: 64
last_iter: 1535
last_upload: 5529600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108
The child process has been launched with process id: 3949491
Executing the command: /var/lib/boinc-client/slots/9/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3949491:3949491] [20230125:224132:1674711692:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:42:13 STEP 1536 H=1536:00 +CPU= 37.511
  22:42:34 STEP 1537 H=1537:00 +CPU= 20.023
  22:42:55 STEP 1538 H=1538:00 +CPU= 21.150
  22:43:28 STEP 1539 H=1539:00 +CPU= 32.666
  22:43:51 STEP 1540 H=1540:00 +CPU= 23.069
  22:44:12 STEP 1541 H=1541:00 +CPU= 20.955
  22:44:45 STEP 1542 H=1542:00 +CPU= 32.736
  22:45:05 STEP 1543 H=1543:00 +CPU= 19.500
  22:45:25 STEP 1544 H=1544:00 +CPU= 19.777
  22:45:57 STEP 1545 H=1545:00 +CPU= 31.785
  22:46:18 STEP 1546 H=1546:00 +CPU= 21.456
  22:46:39 STEP 1547 H=1547:00 +CPU= 20.859
  22:47:14 STEP 1548 H=1548:00 +CPU= 34.538
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001548
  22:47:37 STEP 1549 H=1549:00 +CPU= 22.568
  22:47:57 STEP 1550 H=1550:00 +CPU= 20.708
  22:48:28 STEP 1551 H=1551:00 +CPU= 30.419
  22:48:51 STEP 1552 H=1552:00 +CPU= 22.972
  22:49:13 STEP 1553 H=1553:00 +CPU= 21.506
  22:49:44 STEP 1554 H=1554:00 +CPU= 31.235
  22:50:05 STEP 1555 H=1555:00 +CPU= 20.338
  22:50:27 STEP 1556 H=1556:00 +CPU= 22.658
  22:50:57 STEP 1557 H=1557:00 +CPU= 29.532
  22:51:18 STEP 1558 H=1558:00 +CPU= 20.684
  22:51:41 STEP 1559 H=1559:00 +CPU= 22.506
  22:52:18 STEP 1560 H=1560:00 +CPU= 36.714
 SMILAG TRAJECTORY UNDERGROUND      1 TIMES.
  POINT=           12  MAX ETADOT VERTICAL VEL.=   7.596682938045670E-008
  LON  =    11.2500000000000       degrees
  LAT  =   -45.4202786265483       degrees
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/9/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12189752/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0108_2004050100_123_973_12189752_1_r14159108_64.zip
Uploading the intermediate file: upload_file_64.zip
  22:52:38 STEP 1561 H=1561:00 +CPU= 19.935
  22:52:59 STEP 1562 H=1562:00 +CPU= 21.238
  22:53:34 STEP 1563 H=1563:00 +CPU= 34.108
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:41:14 1532237 22303609 oifs_43r3_ps_0108_2004050100_123_973_12189752_1 1,728,000 10,825 0.0063
09 May 2023 16:41:11 1532237 22303609 oifs_43r3_ps_0108_2004050100_123_973_12189752_1 864,000 5,412 0.0063


©2024 climateprediction.net