climateprediction.net home page
Task 22303608

Task 22303608

Name oifs_43r3_ps_0600_2001050100_123_970_12187244_1
Workunit 12187244
Created 25 Jan 2023, 18:10:28 UTC
Sent 25 Jan 2023, 18:12:37 UTC
Report deadline 26 Mar 2023, 18:12:37 UTC
Received 26 Jan 2023, 7:48:31 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1532237
Run time 11 hours 42 min 32 sec
CPU time 11 hours 31 min 35 sec
Validate state Invalid
Credit 2,419.20
Device peak FLOPS 7.29 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,408.84 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,674.28 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
8 STEP 1292 H=1292:00 +CPU= 21.873
  20:25:51 STEP 1293 H=1293:00 +CPU= 33.124
  20:26:15 STEP 1294 H=1294:00 +CPU= 23.212
  20:26:35 STEP 1295 H=1295:00 +CPU= 20.664
  20:27:13 STEP 1296 H=1296:00 +CPU= 36.832
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_53.zip
Uploading the intermediate file: upload_file_53.zip
  20:27:36 STEP 1297 H=1297:00 +CPU= 23.062
  20:27:56 STEP 1298 H=1298:00 +CPU= 20.062
  20:28:27 STEP 1299 H=1299:00 +CPU= 30.238
  20:28:50 STEP 1300 H=1300:00 +CPU= 23.432
  20:29:09 STEP 1301 H=1301:00 +CPU= 18.719
  20:29:40 STEP 1302 H=1302:00 +CPU= 30.278
  20:30:02 STEP 1303 H=1303:00 +CPU= 21.824
  20:30:25 STEP 1304 H=1304:00 +CPU= 23.529
  20:30:59 STEP 1305 H=1305:00 +CPU= 33.135
  20:31:21 STEP 1306 H=1306:00 +CPU= 22.245
  20:31:42 STEP 1307 H=1307:00 +CPU= 20.772
  20:32:17 STEP 1308 H=1308:00 +CPU= 34.511
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001308
  20:32:39 STEP 1309 H=1309:00 +CPU= 21.817
  20:33:01 STEP 1310 H=1310:00 +CPU= 21.916
  20:33:34 STEP 1311 H=1311:00 +CPU= 32.271
  20:33:57 STEP 1312 H=1312:00 +CPU= 22.896
  20:34:20 STEP 1313 H=1313:00 +CPU= 22.866
  20:34:53 STEP 1314 H=1314:00 +CPU= 33.089
  20:35:14 STEP 1315 H=1315:00 +CPU= 20.387
  20:35:35 STEP 1316 H=1316:00 +CPU= 20.917
  20:36:10 STEP 1317 H=1317:00 +CPU= 34.528
  20:36:33 STEP 1318 H=1318:00 +CPU= 23.154
  20:36:55 STEP 1319 H=1319:00 +CPU= 21.903
  20:37:34 STEP 1320 H=1320:00 +CPU= 38.556
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_54.zip
Uploading the intermediate file: upload_file_54.zip
  20:37:58 STEP 1321 H=1321:00 +CPU= 23.858
  20:38:18 STEP 1322 H=1322:00 +CPU= 20.321
  20:38:51 STEP 1323 H=1323:00 +CPU= 31.945
  20:39:14 STEP 1324 H=1324:00 +CPU= 22.781
  20:39:37 STEP 1325 H=1325:00 +CPU= 22.613
  20:40:11 STEP 1326 H=1326:00 +CPU= 34.293
  20:40:33 STEP 1327 H=1327:00 +CPU= 21.454
  20:40:55 STEP 1328 H=1328:00 +CPU= 21.991
  20:41:28 STEP 1329 H=1329:00 +CPU= 33.223
  20:41:49 STEP 1330 H=1330:00 +CPU= 21.029
  20:42:10 STEP 1331 H=1331:00 +CPU= 20.359
  20:42:45 STEP 1332 H=1332:00 +CPU= 35.137
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001332
  20:43:06 STEP 1333 H=1333:00 +CPU= 20.554
  20:43:27 STEP 1334 H=1334:00 +CPU= 21.230
  20:44:01 STEP 1335 H=1335:00 +CPU= 33.721
  20:44:22 STEP 1336 H=1336:00 +CPU= 20.601
  20:44:44 STEP 1337 H=1337:00 +CPU= 21.930
  20:45:17 STEP 1338 H=1338:00 +CPU= 33.155
  20:45:39 STEP 1339 H=1339:00 +CPU= 21.823
  20:46:03 STEP 1340 H=1340:00 +CPU= 23.344
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0600
(argv4) batchid: 970
(argv5) wuid: 12187244
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_49a62b78f33c8eff361f6a3a0a16d464 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0600_2001050100_123_970_12187244.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0600_2001050100_123_970_12187244.zip
ic_ancil_file: ic_ancil_12187244
ifsdata_file: ifsdata_12187244
climate_data_file: clim_data_12187244
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_59ea3c8daa94d8370a8d527cf5808c72 to: /var/lib/boinc-client/slots/4/ic_ancil_12187244.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12187244.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bb96c4f682936be2ea42ccf3ea8e7dcb to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12187244.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12187244.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12187244.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12187244.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12187244.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12187244.xml
-- Model is restarting --
Adjusting last_iter, 1341, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12187244.xml
last_cpu_time: 33520
upload_file_number: 55
last_iter: 1319
last_upload: 4752000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056
The child process has been launched with process id: 3930456
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3930456:3930456] [20230125:204711:1674704831:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:47:50 STEP 1320 H=1320:00 +CPU= 34.794
  20:48:11 STEP 1321 H=1321:00 +CPU= 20.881
  20:48:34 STEP 1322 H=1322:00 +CPU= 23.278
  20:49:08 STEP 1323 H=1323:00 +CPU= 33.399
  20:49:29 STEP 1324 H=1324:00 +CPU= 21.280
  20:49:50 STEP 1325 H=1325:00 +CPU= 20.886
  20:50:25 STEP 1326 H=1326:00 +CPU= 34.250
  20:50:45 STEP 1327 H=1327:00 +CPU= 20.238
  20:51:06 STEP 1328 H=1328:00 +CPU= 20.780
  20:51:39 STEP 1329 H=1329:00 +CPU= 31.867
  20:52:00 STEP 1330 H=1330:00 +CPU= 21.641
  20:52:23 STEP 1331 H=1331:00 +CPU= 22.320
  20:52:59 STEP 1332 H=1332:00 +CPU= 35.558
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001332
  20:53:21 STEP 1333 H=1333:00 +CPU= 22.154
  20:53:45 STEP 1334 H=1334:00 +CPU= 23.514
  20:54:15 STEP 1335 H=1335:00 +CPU= 30.462
  20:54:36 STEP 1336 H=1336:00 +CPU= 20.534
  20:55:00 STEP 1337 H=1337:00 +CPU= 23.691
  20:55:32 STEP 1338 H=1338:00 +CPU= 31.261
  20:55:51 STEP 1339 H=1339:00 +CPU= 19.232
  20:56:13 STEP 1340 H=1340:00 +CPU= 22.071
  20:56:47 STEP 1341 H=1341:00 +CPU= 33.471
  20:57:08 STEP 1342 H=1342:00 +CPU= 21.021
  20:57:30 STEP 1343 H=1343:00 +CPU= 21.862
  20:58:10 STEP 1344 H=1344:00 +CPU= 39.100
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_55.zip
Uploading the intermediate file: upload_file_55.zip
  20:58:30 STEP 1345 H=1345:00 +CPU= 20.033
  20:58:51 STEP 1346 H=1346:00 +CPU= 20.791
  20:59:24 STEP 1347 H=1347:00 +CPU= 33.060
  20:59:47 STEP 1348 H=1348:00 +CPU= 22.238
  21:00:08 STEP 1349 H=1349:00 +CPU= 21.037
  21:00:42 STEP 1350 H=1350:00 +CPU= 33.557
  21:01:03 STEP 1351 H=1351:00 +CPU= 21.018
  21:01:23 STEP 1352 H=1352:00 +CPU= 20.251
  21:01:55 STEP 1353 H=1353:00 +CPU= 31.245
  21:02:18 STEP 1354 H=1354:00 +CPU= 23.134
  21:02:39 STEP 1355 H=1355:00 +CPU= 20.929
  21:03:14 STEP 1356 H=1356:00 +CPU= 34.832
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001356
  21:03:38 STEP 1357 H=1357:00 +CPU= 22.954
  21:04:00 STEP 1358 H=1358:00 +CPU= 22.263
  21:04:31 STEP 1359 H=1359:00 +CPU= 30.294
  21:04:53 STEP 1360 H=1360:00 +CPU= 22.273
  21:05:18 STEP 1361 H=1361:00 +CPU= 24.265
  21:05:49 STEP 1362 H=1362:00 +CPU= 31.644
  21:06:10 STEP 1363 H=1363:00 +CPU= 20.576
  21:06:34 STEP 1364 H=1364:00 +CPU= 23.470
  21:07:06 STEP 1365 H=1365:00 +CPU= 31.762
  21:07:27 STEP 1366 H=1366:00 +CPU= 20.823
  21:07:51 STEP 1367 H=1367:00 +CPU= 23.646
  21:08:28 STEP 1368 H=1368:00 +CPU= 36.908
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_56.zip
Uploading the intermediate file: upload_file_56.zip
  21:08:47 STEP 1369 H=1369:00 +CPU= 18.510
  21:09:08 STEP 1370 H=1370:00 +CPU= 20.627
  21:09:38 STEP 1371 H=1371:00 +CPU= 30.130
  21:10:00 STEP 1372 H=1372:00 +CPU= 21.589
  21:10:20 STEP 1373 H=1373:00 +CPU= 19.769
  21:10:54 STEP 1374 H=1374:00 +CPU= 33.538
  21:11:14 STEP 1375 H=1375:00 +CPU= 19.883
  21:11:36 STEP 1376 H=1376:00 +CPU= 21.785
  21:12:09 STEP 1377 H=1377:00 +CPU= 33.024
  21:12:32 STEP 1378 H=1378:00 +CPU= 22.921
  21:12:55 STEP 1379 H=1379:00 +CPU= 22.217
  21:13:30 STEP 1380 H=1380:00 +CPU= 35.271
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001380
  21:13:52 STEP 1381 H=1381:00 +CPU= 22.065
  21:14:14 STEP 1382 H=1382:00 +CPU= 21.131
  21:14:46 STEP 1383 H=1383:00 +CPU= 31.807
  21:15:09 STEP 1384 H=1384:00 +CPU= 23.162
  21:15:30 STEP 1385 H=1385:00 +CPU= 21.028
  21:16:06 STEP 1386 H=1386:00 +CPU= 34.794
  21:16:27 STEP 1387 H=1387:00 +CPU= 21.147
  21:16:49 STEP 1388 H=1388:00 +CPU= 21.683
  21:17:22 STEP 1389 H=1389:00 +CPU= 32.953
  21:17:44 STEP 1390 H=1390:00 +CPU= 22.016
  21:18:07 STEP 1391 H=1391:00 +CPU= 22.281
  21:18:45 STEP 1392 H=1392:00 +CPU= 37.805
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_57.zip
Uploading the intermediate file: upload_file_57.zip
  21:19:07 STEP 1393 H=1393:00 +CPU= 22.165
  21:19:29 STEP 1394 H=1394:00 +CPU= 21.572
  21:20:00 STEP 1395 H=1395:00 +CPU= 30.867
  21:20:24 STEP 1396 H=1396:00 +CPU= 23.328
  21:20:47 STEP 1397 H=1397:00 +CPU= 22.401
  21:21:17 STEP 1398 H=1398:00 +CPU= 30.568
  21:21:38 STEP 1399 H=1399:00 +CPU= 20.989
  21:22:02 STEP 1400 H=1400:00 +CPU= 23.184
  21:22:34 STEP 1401 H=1401:00 +CPU= 32.447
  21:22:55 STEP 1402 H=1402:00 +CPU= 20.086
  21:23:14 STEP 1403 H=1403:00 +CPU= 19.340
  21:23:49 STEP 1404 H=1404:00 +CPU= 34.338
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001404
  21:24:11 STEP 1405 H=1405:00 +CPU= 22.108
  21:24:33 STEP 1406 H=1406:00 +CPU= 21.505
  21:25:03 STEP 1407 H=1407:00 +CPU= 30.329
  21:25:27 STEP 1408 H=1408:00 +CPU= 23.373
  21:25:50 STEP 1409 H=1409:00 +CPU= 23.033
  21:26:22 STEP 1410 H=1410:00 +CPU= 31.519
  21:26:44 STEP 1411 H=1411:00 +CPU= 21.826
  21:27:08 STEP 1412 H=1412:00 +CPU= 23.645
  21:27:41 STEP 1413 H=1413:00 +CPU= 33.342
  21:28:03 STEP 1414 H=1414:00 +CPU= 21.485
  21:28:26 STEP 1415 H=1415:00 +CPU= 22.489
  21:29:05 STEP 1416 H=1416:00 +CPU= 39.385
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_58.zip
Uploading the intermediate file: upload_file_58.zip
  21:29:26 STEP 1417 H=1417:00 +CPU= 20.681
  21:29:47 STEP 1418 H=1418:00 +CPU= 20.903
  21:30:23 STEP 1419 H=1419:00 +CPU= 34.984
  21:30:45 STEP 1420 H=1420:00 +CPU= 21.884
  21:31:05 STEP 1421 H=1421:00 +CPU= 19.710
  21:31:39 STEP 1422 H=1422:00 +CPU= 33.937
  21:32:02 STEP 1423 H=1423:00 +CPU= 22.462
  21:32:23 STEP 1424 H=1424:00 +CPU= 20.771
  21:32:54 STEP 1425 H=1425:00 +CPU= 31.228
  21:33:18 STEP 1426 H=1426:00 +CPU= 23.626
  21:33:40 STEP 1427 H=1427:00 +CPU= 22.163
  21:34:16 STEP 1428 H=1428:00 +CPU= 35.370
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001428
  21:34:40 STEP 1429 H=1429:00 +CPU= 23.335
  21:35:03 STEP 1430 H=1430:00 +CPU= 22.721
  21:35:33 STEP 1431 H=1431:00 +CPU= 29.874
  21:35:56 STEP 1432 H=1432:00 +CPU= 22.620
  21:36:19 STEP 1433 H=1433:00 +CPU= 23.416
  21:36:49 STEP 1434 H=1434:00 +CPU= 29.684
  21:37:09 STEP 1435 H=1435:00 +CPU= 19.961
  21:37:34 STEP 1436 H=1436:00 +CPU= 24.026
  21:38:08 STEP 1437 H=1437:00 +CPU= 34.015
  21:38:29 STEP 1438 H=1438:00 +CPU= 20.385
  21:38:50 STEP 1439 H=1439:00 +CPU= 21.588
  21:39:32 STEP 1440 H=1440:00 +CPU= 40.729
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_59.zip
Uploading the intermediate file: upload_file_59.zip
  21:39:55 STEP 1441 H=1441:00 +CPU= 22.904
  21:40:17 STEP 1442 H=1442:00 +CPU= 21.941
  21:40:49 STEP 1443 H=1443:00 +CPU= 32.101
  21:41:11 STEP 1444 H=1444:00 +CPU= 21.293
  21:41:33 STEP 1445 H=1445:00 +CPU= 22.517
  21:42:07 STEP 1446 H=1446:00 +CPU= 33.081
  21:42:28 STEP 1447 H=1447:00 +CPU= 21.139
  21:42:50 STEP 1448 H=1448:00 +CPU= 21.623
  21:43:23 STEP 1449 H=1449:00 +CPU= 33.255
  21:43:47 STEP 1450 H=1450:00 +CPU= 23.302
  21:44:08 STEP 1451 H=1451:00 +CPU= 20.821
  21:44:42 STEP 1452 H=1452:00 +CPU= 34.760
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001452
  21:45:03 STEP 1453 H=1453:00 +CPU= 20.383
  21:45:27 STEP 1454 H=1454:00 +CPU= 23.759
  21:46:01 STEP 1455 H=1455:00 +CPU= 33.918
  21:46:24 STEP 1456 H=1456:00 +CPU= 22.563
  21:46:45 STEP 1457 H=1457:00 +CPU= 21.409
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0600
(argv4) batchid: 970
(argv5) wuid: 12187244
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_49a62b78f33c8eff361f6a3a0a16d464 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0600_2001050100_123_970_12187244.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0600_2001050100_123_970_12187244.zip
ic_ancil_file: ic_ancil_12187244
ifsdata_file: ifsdata_12187244
climate_data_file: clim_data_12187244
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_59ea3c8daa94d8370a8d527cf5808c72 to: /var/lib/boinc-client/slots/4/ic_ancil_12187244.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12187244.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bb96c4f682936be2ea42ccf3ea8e7dcb to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12187244.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12187244.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12187244.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12187244.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12187244.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12187244.xml
-- Model is restarting --
Adjusting last_iter, 1458, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12187244.xml
last_cpu_time: 37078
upload_file_number: 60
last_iter: 1439
last_upload: 5184000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056
The child process has been launched with process id: 3940932
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3940932:3940932] [20230125:214812:1674708492:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:48:53 STEP 1440 H=1440:00 +CPU= 36.865
  21:49:15 STEP 1441 H=1441:00 +CPU= 22.161
  21:49:38 STEP 1442 H=1442:00 +CPU= 22.322
  21:50:09 STEP 1443 H=1443:00 +CPU= 31.380
  21:50:29 STEP 1444 H=1444:00 +CPU= 20.162
  21:50:50 STEP 1445 H=1445:00 +CPU= 20.652
  21:51:23 STEP 1446 H=1446:00 +CPU= 32.392
  21:51:44 STEP 1447 H=1447:00 +CPU= 21.253
  21:52:05 STEP 1448 H=1448:00 +CPU= 20.744
  21:52:39 STEP 1449 H=1449:00 +CPU= 33.493
  21:53:00 STEP 1450 H=1450:00 +CPU= 21.399
  21:53:21 STEP 1451 H=1451:00 +CPU= 20.425
  21:53:58 STEP 1452 H=1452:00 +CPU= 37.007
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001452
  21:54:19 STEP 1453 H=1453:00 +CPU= 20.840
  21:54:39 STEP 1454 H=1454:00 +CPU= 20.037
  21:55:13 STEP 1455 H=1455:00 +CPU= 33.009
  21:55:36 STEP 1456 H=1456:00 +CPU= 22.255
  21:55:57 STEP 1457 H=1457:00 +CPU= 21.068
  21:56:31 STEP 1458 H=1458:00 +CPU= 33.845
  21:56:55 STEP 1459 H=1459:00 +CPU= 23.501
  21:57:17 STEP 1460 H=1460:00 +CPU= 22.654
  21:57:50 STEP 1461 H=1461:00 +CPU= 32.014
  21:58:12 STEP 1462 H=1462:00 +CPU= 22.002
  21:58:35 STEP 1463 H=1463:00 +CPU= 23.194
  21:59:14 STEP 1464 H=1464:00 +CPU= 38.287
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_60.zip
Uploading the intermediate file: upload_file_60.zip
  21:59:37 STEP 1465 H=1465:00 +CPU= 22.382
  21:59:58 STEP 1466 H=1466:00 +CPU= 21.502
  22:00:32 STEP 1467 H=1467:00 +CPU= 33.438
  22:00:54 STEP 1468 H=1468:00 +CPU= 22.138
  22:01:15 STEP 1469 H=1469:00 +CPU= 20.507
  22:01:49 STEP 1470 H=1470:00 +CPU= 33.430
  22:02:10 STEP 1471 H=1471:00 +CPU= 21.435
  22:02:33 STEP 1472 H=1472:00 +CPU= 22.182
  22:03:06 STEP 1473 H=1473:00 +CPU= 32.686
  22:03:26 STEP 1474 H=1474:00 +CPU= 20.636
  22:03:45 STEP 1475 H=1475:00 +CPU= 18.499
  22:04:20 STEP 1476 H=1476:00 +CPU= 34.662
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001476
  22:04:42 STEP 1477 H=1477:00 +CPU= 21.958
  22:05:03 STEP 1478 H=1478:00 +CPU= 20.960
  22:05:36 STEP 1479 H=1479:00 +CPU= 32.515
  22:06:00 STEP 1480 H=1480:00 +CPU= 23.038
  22:06:20 STEP 1481 H=1481:00 +CPU= 20.465
  22:06:54 STEP 1482 H=1482:00 +CPU= 33.396
  22:07:16 STEP 1483 H=1483:00 +CPU= 21.605
  22:07:39 STEP 1484 H=1484:00 +CPU= 23.342
  22:08:12 STEP 1485 H=1485:00 +CPU= 32.267
  22:08:32 STEP 1486 H=1486:00 +CPU= 19.777
  22:08:53 STEP 1487 H=1487:00 +CPU= 21.006
  22:09:33 STEP 1488 H=1488:00 +CPU= 39.094
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_61.zip
Uploading the intermediate file: upload_file_61.zip
  22:09:54 STEP 1489 H=1489:00 +CPU= 21.340
  22:10:15 STEP 1490 H=1490:00 +CPU= 21.151
  22:10:48 STEP 1491 H=1491:00 +CPU= 32.566
  22:11:12 STEP 1492 H=1492:00 +CPU= 23.757
  22:11:35 STEP 1493 H=1493:00 +CPU= 22.288
  22:12:08 STEP 1494 H=1494:00 +CPU= 33.684
  22:12:29 STEP 1495 H=1495:00 +CPU= 19.924
  22:12:50 STEP 1496 H=1496:00 +CPU= 21.533
  22:13:24 STEP 1497 H=1497:00 +CPU= 33.059
  22:13:45 STEP 1498 H=1498:00 +CPU= 21.216
  22:14:05 STEP 1499 H=1499:00 +CPU= 20.269
  22:14:42 STEP 1500 H=1500:00 +CPU= 36.289
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001500
  22:15:02 STEP 1501 H=1501:00 +CPU= 19.683
  22:15:24 STEP 1502 H=1502:00 +CPU= 21.783
  22:15:59 STEP 1503 H=1503:00 +CPU= 34.567
  22:16:20 STEP 1504 H=1504:00 +CPU= 20.942
  22:16:39 STEP 1505 H=1505:00 +CPU= 19.063
  22:17:13 STEP 1506 H=1506:00 +CPU= 33.004
  22:17:36 STEP 1507 H=1507:00 +CPU= 23.555
  22:17:57 STEP 1508 H=1508:00 +CPU= 20.820
  22:18:30 STEP 1509 H=1509:00 +CPU= 31.992
  22:18:55 STEP 1510 H=1510:00 +CPU= 25.350
  22:19:18 STEP 1511 H=1511:00 +CPU= 22.600
  22:19:55 STEP 1512 H=1512:00 +CPU= 36.520
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_62.zip
Uploading the intermediate file: upload_file_62.zip
  22:20:17 STEP 1513 H=1513:00 +CPU= 22.115
  22:20:40 STEP 1514 H=1514:00 +CPU= 22.497
  22:21:11 STEP 1515 H=1515:00 +CPU= 30.906
  22:21:34 STEP 1516 H=1516:00 +CPU= 22.211
  22:21:57 STEP 1517 H=1517:00 +CPU= 23.351
  22:22:32 STEP 1518 H=1518:00 +CPU= 34.328
  22:22:53 STEP 1519 H=1519:00 +CPU= 21.142
  22:23:15 STEP 1520 H=1520:00 +CPU= 21.782
  22:23:49 STEP 1521 H=1521:00 +CPU= 34.111
  22:24:10 STEP 1522 H=1522:00 +CPU= 20.464
  22:24:29 STEP 1523 H=1523:00 +CPU= 19.262
  22:25:06 STEP 1524 H=1524:00 +CPU= 36.623
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001524
  22:25:27 STEP 1525 H=1525:00 +CPU= 20.709
  22:25:48 STEP 1526 H=1526:00 +CPU= 21.083
  22:26:23 STEP 1527 H=1527:00 +CPU= 33.682
  22:26:44 STEP 1528 H=1528:00 +CPU= 21.405
  22:27:05 STEP 1529 H=1529:00 +CPU= 20.321
  22:27:38 STEP 1530 H=1530:00 +CPU= 32.694
  22:28:00 STEP 1531 H=1531:00 +CPU= 22.361
  22:28:22 STEP 1532 H=1532:00 +CPU= 21.581
  22:28:54 STEP 1533 H=1533:00 +CPU= 32.008
  22:29:18 STEP 1534 H=1534:00 +CPU= 23.796
  22:29:41 STEP 1535 H=1535:00 +CPU= 22.372
  22:30:18 STEP 1536 H=1536:00 +CPU= 36.868
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_63.zip
Uploading the intermediate file: upload_file_63.zip
  22:30:36 STEP 1537 H=1537:00 +CPU= 17.721
  22:30:56 STEP 1538 H=1538:00 +CPU= 20.281
  22:31:28 STEP 1539 H=1539:00 +CPU= 31.647
  22:31:51 STEP 1540 H=1540:00 +CPU= 22.210
  22:32:11 STEP 1541 H=1541:00 +CPU= 20.280
  22:32:43 STEP 1542 H=1542:00 +CPU= 31.573
  22:33:05 STEP 1543 H=1543:00 +CPU= 22.023
  22:33:29 STEP 1544 H=1544:00 +CPU= 23.705
  22:33:58 STEP 1545 H=1545:00 +CPU= 28.810
  22:34:20 STEP 1546 H=1546:00 +CPU= 22.084
  22:34:45 STEP 1547 H=1547:00 +CPU= 24.095
  22:35:20 STEP 1548 H=1548:00 +CPU= 35.228
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001548
  22:35:42 STEP 1549 H=1549:00 +CPU= 21.212
  22:36:04 STEP 1550 H=1550:00 +CPU= 22.554
  22:36:37 STEP 1551 H=1551:00 +CPU= 32.134
  22:37:00 STEP 1552 H=1552:00 +CPU= 22.292
  22:37:23 STEP 1553 H=1553:00 +CPU= 23.249
  22:37:58 STEP 1554 H=1554:00 +CPU= 34.359
  22:38:19 STEP 1555 H=1555:00 +CPU= 21.751
  22:38:43 STEP 1556 H=1556:00 +CPU= 23.024
  22:39:17 STEP 1557 H=1557:00 +CPU= 33.379
  22:39:38 STEP 1558 H=1558:00 +CPU= 20.905
  22:40:00 STEP 1559 H=1559:00 +CPU= 22.348
  22:40:40 STEP 1560 H=1560:00 +CPU= 39.521
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_64.zip
Uploading the intermediate file: upload_file_64.zip
  22:41:00 STEP 1561 H=1561:00 +CPU= 20.155
  22:41:21 STEP 1562 H=1562:00 +CPU= 20.627
  22:41:57 STEP 1563 H=1563:00 +CPU= 34.825
  22:42:18 STEP 1564 H=1564:00 +CPU= 21.764
  22:42:39 STEP 1565 H=1565:00 +CPU= 20.850
  22:43:13 STEP 1566 H=1566:00 +CPU= 33.439
  22:43:35 STEP 1567 H=1567:00 +CPU= 21.534
  22:43:57 STEP 1568 H=1568:00 +CPU= 21.794
  22:44:29 STEP 1569 H=1569:00 +CPU= 32.125
  22:44:50 STEP 1570 H=1570:00 +CPU= 20.419
  22:45:09 STEP 1571 H=1571:00 +CPU= 19.332
  22:45:41 STEP 1572 H=1572:00 +CPU= 32.058
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001572
  22:46:03 STEP 1573 H=1573:00 +CPU= 21.847
  22:46:25 STEP 1574 H=1574:00 +CPU= 21.583
  22:46:55 STEP 1575 H=1575:00 +CPU= 29.387
  22:47:17 STEP 1576 H=1576:00 +CPU= 22.403
  22:47:39 STEP 1577 H=1577:00 +CPU= 21.813
  22:48:11 STEP 1578 H=1578:00 +CPU= 31.815
  22:48:32 STEP 1579 H=1579:00 +CPU= 20.356
  22:48:55 STEP 1580 H=1580:00 +CPU= 22.998
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2001050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0600
(argv4) batchid: 970
(argv5) wuid: 12187244
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_49a62b78f33c8eff361f6a3a0a16d464 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0600_2001050100_123_970_12187244.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0600_2001050100_123_970_12187244.zip
ic_ancil_file: ic_ancil_12187244
ifsdata_file: ifsdata_12187244
climate_data_file: clim_data_12187244
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_59ea3c8daa94d8370a8d527cf5808c72 to: /var/lib/boinc-client/slots/4/ic_ancil_12187244.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12187244.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_bb96c4f682936be2ea42ccf3ea8e7dcb to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12187244.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12187244.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12187244.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12187244.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12187244.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12187244.xml
-- Model is restarting --
Adjusting last_iter, 1580, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12187244.xml
last_cpu_time: 40686
upload_file_number: 65
last_iter: 1559
last_upload: 5616000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056
The child process has been launched with process id: 3951411
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:wkst-01.kalnet.xyz:1:1:3951411:3951411] [20230125:225005:1674712205:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:50:46 STEP 1560 H=1560:00 +CPU= 36.855
  22:51:06 STEP 1561 H=1561:00 +CPU= 19.790
  22:51:27 STEP 1562 H=1562:00 +CPU= 20.941
  22:52:01 STEP 1563 H=1563:00 +CPU= 34.086
  22:52:22 STEP 1564 H=1564:00 +CPU= 20.124
  22:52:42 STEP 1565 H=1565:00 +CPU= 20.075
  22:53:14 STEP 1566 H=1566:00 +CPU= 32.475
  22:53:37 STEP 1567 H=1567:00 +CPU= 22.544
  22:53:58 STEP 1568 H=1568:00 +CPU= 21.031
  22:54:29 STEP 1569 H=1569:00 +CPU= 30.404
  22:54:50 STEP 1570 H=1570:00 +CPU= 20.976
  22:55:10 STEP 1571 H=1571:00 +CPU= 20.189
  22:55:43 STEP 1572 H=1572:00 +CPU= 32.501
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001572
  22:56:02 STEP 1573 H=1573:00 +CPU= 19.089
  22:56:22 STEP 1574 H=1574:00 +CPU= 19.235
  22:56:51 STEP 1575 H=1575:00 +CPU= 28.992
  22:57:12 STEP 1576 H=1576:00 +CPU= 21.027
  22:57:32 STEP 1577 H=1577:00 +CPU= 19.703
  22:58:02 STEP 1578 H=1578:00 +CPU= 30.134
  22:58:20 STEP 1579 H=1579:00 +CPU= 18.013
  22:58:41 STEP 1580 H=1580:00 +CPU= 19.965
  22:59:02 STEP 1581 H=1581:00 +CPU= 21.363
  22:59:16 STEP 1582 H=1582:00 +CPU= 14.470
  22:59:31 STEP 1583 H=1583:00 +CPU= 14.211
  22:59:57 STEP 1584 H=1584:00 +CPU= 25.959
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12187244/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0600_2001050100_123_970_12187244_1_r1558955056_65.zip
Uploading the intermediate file: upload_file_65.zip
  23:00:10 STEP 1585 H=1585:00 +CPU= 13.488
  23:00:25 STEP 1586 H=1586:00 +CPU= 14.321
  23:00:46 STEP 1587 H=1587:00 +CPU= 20.985
  23:01:00 STEP 1588 H=1588:00 +CPU= 14.073
  23:01:14 STEP 1589 H=1589:00 +CPU= 13.868
  23:01:36 STEP 1590 H=1590:00 +CPU= 22.222
  23:01:50 STEP 1591 H=1591:00 +CPU= 13.819
  23:02:04 STEP 1592 H=1592:00 +CPU= 13.928
  23:02:26 STEP 1593 H=1593:00 +CPU= 21.583
  23:02:40 STEP 1594 H=1594:00 +CPU= 13.768
  23:02:53 STEP 1595 H=1595:00 +CPU= 13.476
  23:03:17 STEP 1596 H=1596:00 +CPU= 23.480
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001596
  23:03:31 STEP 1597 H=1597:00 +CPU= 14.248
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:41:14 1532237 22303608 oifs_43r3_ps_0600_2001050100_123_970_12187244_1 1,728,000 10,815 0.0063
09 May 2023 16:41:11 1532237 22303608 oifs_43r3_ps_0600_2001050100_123_970_12187244_1 864,000 5,412 0.0063


©2024 climateprediction.net