climateprediction.net home page
Task 22273553

Task 22273553

Name oifs_43r3_ps_0440_2002050100_123_971_12188084_0
Workunit 12188084
Created 23 Dec 2022, 2:44:31 UTC
Sent 4 Jan 2023, 5:33:18 UTC
Report deadline 3 Feb 2023, 5:33:18 UTC
Received 5 Jan 2023, 7:26:17 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1528008
Run time 15 hours 22 min 23 sec
CPU time 12 hours 28 min 24 sec
Validate state Invalid
Credit 6,289.92
Device peak FLOPS 5.54 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,302.72 MB
Peak swap size 4,978.08 MB
Peak disk usage 7,841.96 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_603411aaa83bb6c83c50f77facc271a0 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12188084.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12188084.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12188084.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12188084.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12188084.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12188084.xml
-- Model is restarting --
Adjusting last_iter, 1337, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12188084.xml
last_cpu_time: 36992
upload_file_number: 55
last_iter: 1319
last_upload: 4752000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189
The child process has been launched with process id: 2137129
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2137129:2137129] [20230104:222906:1672889346:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:29:42 STEP 1320 H=1320:00 +CPU= 31.285
  22:30:03 STEP 1321 H=1321:00 +CPU= 18.099
  22:30:21 STEP 1322 H=1322:00 +CPU= 17.549
  22:30:47 STEP 1323 H=1323:00 +CPU= 25.425
  22:31:04 STEP 1324 H=1324:00 +CPU= 16.486
  22:31:20 STEP 1325 H=1325:00 +CPU= 16.214
  22:31:47 STEP 1326 H=1326:00 +CPU= 26.638
  22:32:04 STEP 1327 H=1327:00 +CPU= 16.414
  22:32:21 STEP 1328 H=1328:00 +CPU= 16.900
  22:32:48 STEP 1329 H=1329:00 +CPU= 26.063
  22:33:05 STEP 1330 H=1330:00 +CPU= 16.899
  22:33:28 STEP 1331 H=1331:00 +CPU= 16.388
  22:33:59 STEP 1332 H=1332:00 +CPU= 30.020
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001332
  22:34:18 STEP 1333 H=1333:00 +CPU= 17.813
  22:34:42 STEP 1334 H=1334:00 +CPU= 17.806
  22:35:07 STEP 1335 H=1335:00 +CPU= 25.345
  22:35:24 STEP 1336 H=1336:00 +CPU= 16.422
  22:35:40 STEP 1337 H=1337:00 +CPU= 16.016
  22:36:07 STEP 1338 H=1338:00 +CPU= 26.381
  22:36:23 STEP 1339 H=1339:00 +CPU= 16.245
  22:36:40 STEP 1340 H=1340:00 +CPU= 16.702
  22:37:06 STEP 1341 H=1341:00 +CPU= 25.823
  22:37:23 STEP 1342 H=1342:00 +CPU= 16.720
  22:37:40 STEP 1343 H=1343:00 +CPU= 16.678
  22:38:12 STEP 1344 H=1344:00 +CPU= 30.237
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_55.zip
Uploading the intermediate file: upload_file_55.zip
  22:38:29 STEP 1345 H=1345:00 +CPU= 16.161
  22:38:46 STEP 1346 H=1346:00 +CPU= 16.655
  22:39:12 STEP 1347 H=1347:00 +CPU= 25.729
  22:39:29 STEP 1348 H=1348:00 +CPU= 16.709
  22:39:45 STEP 1349 H=1349:00 +CPU= 16.225
  22:40:12 STEP 1350 H=1350:00 +CPU= 26.374
  22:40:28 STEP 1351 H=1351:00 +CPU= 16.252
  22:40:45 STEP 1352 H=1352:00 +CPU= 16.731
  22:41:12 STEP 1353 H=1353:00 +CPU= 25.801
  22:41:29 STEP 1354 H=1354:00 +CPU= 16.760
  22:41:46 STEP 1355 H=1355:00 +CPU= 16.565
  22:42:14 STEP 1356 H=1356:00 +CPU= 28.486
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001356
  22:42:31 STEP 1357 H=1357:00 +CPU= 16.242
  22:42:48 STEP 1358 H=1358:00 +CPU= 16.726
  22:43:14 STEP 1359 H=1359:00 +CPU= 25.787
  22:43:31 STEP 1360 H=1360:00 +CPU= 16.720
  22:43:48 STEP 1361 H=1361:00 +CPU= 16.253
  22:44:14 STEP 1362 H=1362:00 +CPU= 26.362
  22:44:31 STEP 1363 H=1363:00 +CPU= 16.199
  22:44:48 STEP 1364 H=1364:00 +CPU= 16.705
  22:45:14 STEP 1365 H=1365:00 +CPU= 25.779
  22:45:31 STEP 1366 H=1366:00 +CPU= 16.696
  22:45:48 STEP 1367 H=1367:00 +CPU= 16.676
  22:46:21 STEP 1368 H=1368:00 +CPU= 30.215
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_56.zip
Uploading the intermediate file: upload_file_56.zip
  22:46:38 STEP 1369 H=1369:00 +CPU= 16.218
  22:46:55 STEP 1370 H=1370:00 +CPU= 16.732
  22:47:21 STEP 1371 H=1371:00 +CPU= 25.837
  22:47:38 STEP 1372 H=1372:00 +CPU= 16.750
  22:47:55 STEP 1373 H=1373:00 +CPU= 16.262
  22:48:22 STEP 1374 H=1374:00 +CPU= 26.414
  22:48:38 STEP 1375 H=1375:00 +CPU= 16.275
  22:48:55 STEP 1376 H=1376:00 +CPU= 16.771
  22:49:22 STEP 1377 H=1377:00 +CPU= 25.907
  22:49:39 STEP 1378 H=1378:00 +CPU= 16.771
  22:49:55 STEP 1379 H=1379:00 +CPU= 16.290
  22:50:24 STEP 1380 H=1380:00 +CPU= 28.631
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001380
  22:50:41 STEP 1381 H=1381:00 +CPU= 16.312
  22:50:58 STEP 1382 H=1382:00 +CPU= 16.819
  22:51:24 STEP 1383 H=1383:00 +CPU= 25.882
  22:51:41 STEP 1384 H=1384:00 +CPU= 16.823
  22:51:58 STEP 1385 H=1385:00 +CPU= 16.311
  22:52:25 STEP 1386 H=1386:00 +CPU= 26.481
  22:52:41 STEP 1387 H=1387:00 +CPU= 16.295
  22:52:58 STEP 1388 H=1388:00 +CPU= 16.799
  22:53:24 STEP 1389 H=1389:00 +CPU= 25.920
  22:53:41 STEP 1390 H=1390:00 +CPU= 16.775
  22:53:58 STEP 1391 H=1391:00 +CPU= 16.762
  22:54:29 STEP 1392 H=1392:00 +CPU= 30.326
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_57.zip
Uploading the intermediate file: upload_file_57.zip
  22:54:46 STEP 1393 H=1393:00 +CPU= 16.289
  22:55:03 STEP 1394 H=1394:00 +CPU= 16.772
  22:55:30 STEP 1395 H=1395:00 +CPU= 25.978
  22:55:47 STEP 1396 H=1396:00 +CPU= 16.831
  22:56:03 STEP 1397 H=1397:00 +CPU= 16.315
  22:56:30 STEP 1398 H=1398:00 +CPU= 26.535
  22:56:47 STEP 1399 H=1399:00 +CPU= 16.341
  22:57:04 STEP 1400 H=1400:00 +CPU= 16.883
  22:57:30 STEP 1401 H=1401:00 +CPU= 25.970
  22:57:47 STEP 1402 H=1402:00 +CPU= 16.888
  22:58:04 STEP 1403 H=1403:00 +CPU= 16.421
  22:58:33 STEP 1404 H=1404:00 +CPU= 28.785
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001404
  22:58:50 STEP 1405 H=1405:00 +CPU= 16.393
  22:59:07 STEP 1406 H=1406:00 +CPU= 16.869
  22:59:33 STEP 1407 H=1407:00 +CPU= 26.010
  22:59:50 STEP 1408 H=1408:00 +CPU= 16.872
  23:00:07 STEP 1409 H=1409:00 +CPU= 16.371
  23:00:34 STEP 1410 H=1410:00 +CPU= 26.553
  23:00:51 STEP 1411 H=1411:00 +CPU= 16.366
  23:01:08 STEP 1412 H=1412:00 +CPU= 16.853
  23:01:34 STEP 1413 H=1413:00 +CPU= 26.016
  23:01:51 STEP 1414 H=1414:00 +CPU= 16.855
  23:02:08 STEP 1415 H=1415:00 +CPU= 16.855
  23:02:39 STEP 1416 H=1416:00 +CPU= 30.455
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_58.zip
Uploading the intermediate file: upload_file_58.zip
  23:02:56 STEP 1417 H=1417:00 +CPU= 16.424
  23:03:13 STEP 1418 H=1418:00 +CPU= 16.836
  23:03:40 STEP 1419 H=1419:00 +CPU= 26.042
  23:03:57 STEP 1420 H=1420:00 +CPU= 16.873
  23:04:14 STEP 1421 H=1421:00 +CPU= 16.408
  23:04:40 STEP 1422 H=1422:00 +CPU= 26.623
  23:04:57 STEP 1423 H=1423:00 +CPU= 16.401
  23:05:14 STEP 1424 H=1424:00 +CPU= 16.914
  23:05:41 STEP 1425 H=1425:00 +CPU= 26.139
  23:05:58 STEP 1426 H=1426:00 +CPU= 16.939
  23:06:15 STEP 1427 H=1427:00 +CPU= 16.444
  23:06:44 STEP 1428 H=1428:00 +CPU= 28.823
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001428
  23:07:01 STEP 1429 H=1429:00 +CPU= 16.453
  23:07:18 STEP 1430 H=1430:00 +CPU= 16.921
  23:07:44 STEP 1431 H=1431:00 +CPU= 26.172
  23:08:01 STEP 1432 H=1432:00 +CPU= 16.918
  23:08:18 STEP 1433 H=1433:00 +CPU= 16.427
  23:08:45 STEP 1434 H=1434:00 +CPU= 26.704
  23:09:02 STEP 1435 H=1435:00 +CPU= 16.443
  23:09:19 STEP 1436 H=1436:00 +CPU= 16.902
  23:09:46 STEP 1437 H=1437:00 +CPU= 26.162
  23:10:03 STEP 1438 H=1438:00 +CPU= 16.906
  23:10:20 STEP 1439 H=1439:00 +CPU= 16.953
  23:10:51 STEP 1440 H=1440:00 +CPU= 30.694
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_59.zip
Uploading the intermediate file: upload_file_59.zip
  23:11:08 STEP 1441 H=1441:00 +CPU= 16.442
  23:11:26 STEP 1442 H=1442:00 +CPU= 16.910
  23:11:52 STEP 1443 H=1443:00 +CPU= 26.169
  23:12:09 STEP 1444 H=1444:00 +CPU= 16.944
  23:12:26 STEP 1445 H=1445:00 +CPU= 16.426
  23:12:53 STEP 1446 H=1446:00 +CPU= 26.765
  23:13:10 STEP 1447 H=1447:00 +CPU= 16.480
  23:13:27 STEP 1448 H=1448:00 +CPU= 16.983
  23:13:53 STEP 1449 H=1449:00 +CPU= 26.235
  23:14:11 STEP 1450 H=1450:00 +CPU= 16.963
  23:14:27 STEP 1451 H=1451:00 +CPU= 16.520
  23:14:57 STEP 1452 H=1452:00 +CPU= 28.944
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001452
  23:15:14 STEP 1453 H=1453:00 +CPU= 16.501
  23:15:31 STEP 1454 H=1454:00 +CPU= 16.998
  23:15:57 STEP 1455 H=1455:00 +CPU= 26.231
  23:16:15 STEP 1456 H=1456:00 +CPU= 16.968
  23:16:31 STEP 1457 H=1457:00 +CPU= 16.490
  23:16:58 STEP 1458 H=1458:00 +CPU= 26.789
  23:17:15 STEP 1459 H=1459:00 +CPU= 16.506
  23:17:32 STEP 1460 H=1460:00 +CPU= 16.954
  23:17:59 STEP 1461 H=1461:00 +CPU= 26.231
  23:18:16 STEP 1462 H=1462:00 +CPU= 16.954
  23:18:33 STEP 1463 H=1463:00 +CPU= 16.953
  23:19:05 STEP 1464 H=1464:00 +CPU= 30.775
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_60.zip
Uploading the intermediate file: upload_file_60.zip
  23:19:22 STEP 1465 H=1465:00 +CPU= 16.457
  23:19:39 STEP 1466 H=1466:00 +CPU= 16.966
  23:20:06 STEP 1467 H=1467:00 +CPU= 26.291
  23:20:23 STEP 1468 H=1468:00 +CPU= 16.999
  23:20:40 STEP 1469 H=1469:00 +CPU= 16.527
  23:21:07 STEP 1470 H=1470:00 +CPU= 26.860
  23:21:24 STEP 1471 H=1471:00 +CPU= 16.531
  23:21:41 STEP 1472 H=1472:00 +CPU= 17.019
  23:22:07 STEP 1473 H=1473:00 +CPU= 26.275
  23:22:25 STEP 1474 H=1474:00 +CPU= 17.009
  23:22:41 STEP 1475 H=1475:00 +CPU= 16.511
  23:23:11 STEP 1476 H=1476:00 +CPU= 29.053
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001476
  23:23:28 STEP 1477 H=1477:00 +CPU= 16.550
  23:23:45 STEP 1478 H=1478:00 +CPU= 17.026
  23:24:12 STEP 1479 H=1479:00 +CPU= 26.351
  23:24:29 STEP 1480 H=1480:00 +CPU= 17.045
  23:24:46 STEP 1481 H=1481:00 +CPU= 16.514
  23:25:13 STEP 1482 H=1482:00 +CPU= 26.864
  23:25:30 STEP 1483 H=1483:00 +CPU= 16.498
  23:25:47 STEP 1484 H=1484:00 +CPU= 16.973
  23:26:14 STEP 1485 H=1485:00 +CPU= 26.335
  23:26:31 STEP 1486 H=1486:00 +CPU= 17.038
  23:26:48 STEP 1487 H=1487:00 +CPU= 17.002
  23:27:19 STEP 1488 H=1488:00 +CPU= 30.817
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_61.zip
Uploading the intermediate file: upload_file_61.zip
  23:27:36 STEP 1489 H=1489:00 +CPU= 16.470
  23:27:54 STEP 1490 H=1490:00 +CPU= 16.999
  23:28:20 STEP 1491 H=1491:00 +CPU= 26.303
  23:28:37 STEP 1492 H=1492:00 +CPU= 17.022
  23:28:54 STEP 1493 H=1493:00 +CPU= 16.525
  23:29:21 STEP 1494 H=1494:00 +CPU= 26.868
  23:29:38 STEP 1495 H=1495:00 +CPU= 16.534
  23:29:55 STEP 1496 H=1496:00 +CPU= 17.057
  23:30:22 STEP 1497 H=1497:00 +CPU= 26.338
  23:30:39 STEP 1498 H=1498:00 +CPU= 17.006
  23:30:57 STEP 1499 H=1499:00 +CPU= 17.295
  23:31:30 STEP 1500 H=1500:00 +CPU= 31.195
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001500
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0440
(argv4) batchid: 971
(argv5) wuid: 12188084
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c349c1784683faa42d2f0134579d5c74 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0440_2002050100_123_971_12188084.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0440_2002050100_123_971_12188084.zip
ic_ancil_file: ic_ancil_12188084
ifsdata_file: ifsdata_12188084
climate_data_file: clim_data_12188084
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_494768d76891e3f9dcfef535df354b75 to: /var/lib/boinc-client/slots/6/ic_ancil_12188084.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12188084.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_603411aaa83bb6c83c50f77facc271a0 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12188084.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12188084.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12188084.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12188084.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12188084.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12188084.xml
-- Model is restarting --
Adjusting last_iter, 1501, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12188084.xml
last_cpu_time: 40659
upload_file_number: 62
last_iter: 1487
last_upload: 5356800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189
The child process has been launched with process id: 2141565
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2141565:2141565] [20230104:233542:1672893342:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  23:36:22 STEP 1488 H=1488:00 +CPU= 31.487
  23:36:43 STEP 1489 H=1489:00 +CPU= 18.025
  23:37:00 STEP 1490 H=1490:00 +CPU= 17.116
  23:37:27 STEP 1491 H=1491:00 +CPU= 26.470
  23:37:45 STEP 1492 H=1492:00 +CPU= 17.262
  23:38:08 STEP 1493 H=1493:00 +CPU= 17.991
  23:38:36 STEP 1494 H=1494:00 +CPU= 27.530
  23:38:52 STEP 1495 H=1495:00 +CPU= 16.206
  23:39:09 STEP 1496 H=1496:00 +CPU= 16.655
  23:39:35 STEP 1497 H=1497:00 +CPU= 26.186
  23:39:53 STEP 1498 H=1498:00 +CPU= 17.051
  23:40:10 STEP 1499 H=1499:00 +CPU= 16.588
  23:40:39 STEP 1500 H=1500:00 +CPU= 29.115
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001500
  23:40:56 STEP 1501 H=1501:00 +CPU= 16.610
  23:41:14 STEP 1502 H=1502:00 +CPU= 17.096
  23:41:40 STEP 1503 H=1503:00 +CPU= 26.395
  23:41:58 STEP 1504 H=1504:00 +CPU= 17.093
  23:42:15 STEP 1505 H=1505:00 +CPU= 16.623
  23:42:42 STEP 1506 H=1506:00 +CPU= 26.920
  23:42:59 STEP 1507 H=1507:00 +CPU= 16.618
  23:43:17 STEP 1508 H=1508:00 +CPU= 17.126
  23:43:43 STEP 1509 H=1509:00 +CPU= 26.363
  23:44:01 STEP 1510 H=1510:00 +CPU= 17.075
  23:44:18 STEP 1511 H=1511:00 +CPU= 17.106
  23:44:50 STEP 1512 H=1512:00 +CPU= 30.845
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_62.zip
Uploading the intermediate file: upload_file_62.zip
  23:45:07 STEP 1513 H=1513:00 +CPU= 16.505
  23:45:25 STEP 1514 H=1514:00 +CPU= 16.983
  23:45:51 STEP 1515 H=1515:00 +CPU= 26.269
  23:46:09 STEP 1516 H=1516:00 +CPU= 17.026
  23:46:25 STEP 1517 H=1517:00 +CPU= 16.562
  23:46:53 STEP 1518 H=1518:00 +CPU= 26.850
  23:47:10 STEP 1519 H=1519:00 +CPU= 16.535
  23:47:27 STEP 1520 H=1520:00 +CPU= 17.043
  23:47:54 STEP 1521 H=1521:00 +CPU= 26.268
  23:48:11 STEP 1522 H=1522:00 +CPU= 17.049
  23:48:28 STEP 1523 H=1523:00 +CPU= 16.555
  23:48:57 STEP 1524 H=1524:00 +CPU= 29.066
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001524
  23:49:14 STEP 1525 H=1525:00 +CPU= 16.578
  23:49:32 STEP 1526 H=1526:00 +CPU= 17.038
  23:49:58 STEP 1527 H=1527:00 +CPU= 26.346
  23:50:16 STEP 1528 H=1528:00 +CPU= 17.095
  23:50:33 STEP 1529 H=1529:00 +CPU= 16.682
  23:51:00 STEP 1530 H=1530:00 +CPU= 27.087
  23:51:17 STEP 1531 H=1531:00 +CPU= 16.620
  23:51:34 STEP 1532 H=1532:00 +CPU= 17.062
  23:52:01 STEP 1533 H=1533:00 +CPU= 26.378
  23:52:18 STEP 1534 H=1534:00 +CPU= 17.003
  23:52:36 STEP 1535 H=1535:00 +CPU= 16.971
  23:53:08 STEP 1536 H=1536:00 +CPU= 30.635
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_63.zip
Uploading the intermediate file: upload_file_63.zip
  23:53:25 STEP 1537 H=1537:00 +CPU= 16.510
  23:53:42 STEP 1538 H=1538:00 +CPU= 17.012
  23:54:09 STEP 1539 H=1539:00 +CPU= 26.288
  23:54:26 STEP 1540 H=1540:00 +CPU= 17.047
  23:54:43 STEP 1541 H=1541:00 +CPU= 16.508
  23:55:10 STEP 1542 H=1542:00 +CPU= 26.841
  23:55:27 STEP 1543 H=1543:00 +CPU= 16.529
  23:55:44 STEP 1544 H=1544:00 +CPU= 17.062
  23:56:11 STEP 1545 H=1545:00 +CPU= 26.305
  23:56:28 STEP 1546 H=1546:00 +CPU= 17.069
  23:56:45 STEP 1547 H=1547:00 +CPU= 16.524
  23:57:15 STEP 1548 H=1548:00 +CPU= 29.045
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001548
  23:57:32 STEP 1549 H=1549:00 +CPU= 16.590
  23:57:49 STEP 1550 H=1550:00 +CPU= 17.046
  23:58:16 STEP 1551 H=1551:00 +CPU= 26.269
  23:58:33 STEP 1552 H=1552:00 +CPU= 16.970
  23:58:50 STEP 1553 H=1553:00 +CPU= 16.456
  23:59:17 STEP 1554 H=1554:00 +CPU= 26.804
  23:59:34 STEP 1555 H=1555:00 +CPU= 16.451
  23:59:51 STEP 1556 H=1556:00 +CPU= 16.965
  00:00:18 STEP 1557 H=1557:00 +CPU= 26.290
  00:00:35 STEP 1558 H=1558:00 +CPU= 16.966
  00:00:52 STEP 1559 H=1559:00 +CPU= 16.926
  00:01:23 STEP 1560 H=1560:00 +CPU= 30.429
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_64.zip
Uploading the intermediate file: upload_file_64.zip
  00:01:40 STEP 1561 H=1561:00 +CPU= 16.487
  00:01:58 STEP 1562 H=1562:00 +CPU= 17.011
  00:02:25 STEP 1563 H=1563:00 +CPU= 26.348
  00:02:42 STEP 1564 H=1564:00 +CPU= 16.995
  00:02:59 STEP 1565 H=1565:00 +CPU= 16.520
  00:03:26 STEP 1566 H=1566:00 +CPU= 26.801
  00:03:43 STEP 1567 H=1567:00 +CPU= 16.485
  00:04:00 STEP 1568 H=1568:00 +CPU= 17.002
  00:04:27 STEP 1569 H=1569:00 +CPU= 26.223
  00:04:44 STEP 1570 H=1570:00 +CPU= 16.948
  00:05:00 STEP 1571 H=1571:00 +CPU= 16.496
  00:05:30 STEP 1572 H=1572:00 +CPU= 28.911
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001572
  00:05:47 STEP 1573 H=1573:00 +CPU= 16.472
  00:06:04 STEP 1574 H=1574:00 +CPU= 16.991
  00:06:31 STEP 1575 H=1575:00 +CPU= 26.231
  00:06:48 STEP 1576 H=1576:00 +CPU= 16.971
  00:07:05 STEP 1577 H=1577:00 +CPU= 16.436
  00:07:32 STEP 1578 H=1578:00 +CPU= 26.781
  00:07:48 STEP 1579 H=1579:00 +CPU= 16.480
  00:08:06 STEP 1580 H=1580:00 +CPU= 16.967
  00:08:32 STEP 1581 H=1581:00 +CPU= 26.168
  00:08:49 STEP 1582 H=1582:00 +CPU= 16.931
  00:09:07 STEP 1583 H=1583:00 +CPU= 16.903
  00:09:37 STEP 1584 H=1584:00 +CPU= 30.534
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_65.zip
Uploading the intermediate file: upload_file_65.zip
  00:09:54 STEP 1585 H=1585:00 +CPU= 16.449
  00:10:12 STEP 1586 H=1586:00 +CPU= 16.990
  00:10:38 STEP 1587 H=1587:00 +CPU= 26.292
  00:10:56 STEP 1588 H=1588:00 +CPU= 16.964
  00:11:12 STEP 1589 H=1589:00 +CPU= 16.503
  00:11:40 STEP 1590 H=1590:00 +CPU= 26.828
  00:11:56 STEP 1591 H=1591:00 +CPU= 16.499
  00:12:14 STEP 1592 H=1592:00 +CPU= 16.974
  00:12:40 STEP 1593 H=1593:00 +CPU= 26.153
  00:12:57 STEP 1594 H=1594:00 +CPU= 16.939
  00:13:14 STEP 1595 H=1595:00 +CPU= 16.430
  00:13:44 STEP 1596 H=1596:00 +CPU= 29.674
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001596
  00:14:07 STEP 1597 H=1597:00 +CPU= 17.592
  00:14:27 STEP 1598 H=1598:00 +CPU= 18.198
  00:14:55 STEP 1599 H=1599:00 +CPU= 27.540
  00:15:12 STEP 1600 H=1600:00 +CPU= 16.581
  00:15:28 STEP 1601 H=1601:00 +CPU= 16.064
  00:15:55 STEP 1602 H=1602:00 +CPU= 26.376
  00:16:16 STEP 1603 H=1603:00 +CPU= 17.451
  00:16:34 STEP 1604 H=1604:00 +CPU= 18.264
  00:17:05 STEP 1605 H=1605:00 +CPU= 27.842
  00:17:21 STEP 1606 H=1606:00 +CPU= 16.502
  00:17:38 STEP 1607 H=1607:00 +CPU= 16.519
  00:18:10 STEP 1608 H=1608:00 +CPU= 29.916
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_66.zip
Uploading the intermediate file: upload_file_66.zip
  00:18:27 STEP 1609 H=1609:00 +CPU= 16.335
  00:18:44 STEP 1610 H=1610:00 +CPU= 16.808
  00:19:10 STEP 1611 H=1611:00 +CPU= 25.978
  00:19:27 STEP 1612 H=1612:00 +CPU= 16.792
  00:19:44 STEP 1613 H=1613:00 +CPU= 16.318
  00:20:11 STEP 1614 H=1614:00 +CPU= 26.521
  00:20:27 STEP 1615 H=1615:00 +CPU= 16.329
  00:20:45 STEP 1616 H=1616:00 +CPU= 16.833
  00:21:11 STEP 1617 H=1617:00 +CPU= 25.922
  00:21:28 STEP 1618 H=1618:00 +CPU= 16.816
  00:21:45 STEP 1619 H=1619:00 +CPU= 16.326
  00:22:14 STEP 1620 H=1620:00 +CPU= 28.724
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001620
  00:22:31 STEP 1621 H=1621:00 +CPU= 16.342
  00:22:48 STEP 1622 H=1622:00 +CPU= 16.827
  00:23:14 STEP 1623 H=1623:00 +CPU= 25.980
  00:23:31 STEP 1624 H=1624:00 +CPU= 16.816
  00:23:48 STEP 1625 H=1625:00 +CPU= 16.335
  00:24:15 STEP 1626 H=1626:00 +CPU= 26.502
  00:24:31 STEP 1627 H=1627:00 +CPU= 16.313
  00:24:48 STEP 1628 H=1628:00 +CPU= 16.809
  00:25:15 STEP 1629 H=1629:00 +CPU= 25.983
  00:25:32 STEP 1630 H=1630:00 +CPU= 16.812
  00:25:49 STEP 1631 H=1631:00 +CPU= 16.791
  00:26:20 STEP 1632 H=1632:00 +CPU= 30.200
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_67.zip
Uploading the intermediate file: upload_file_67.zip
  00:26:36 STEP 1633 H=1633:00 +CPU= 16.331
  00:26:53 STEP 1634 H=1634:00 +CPU= 16.798
  00:27:20 STEP 1635 H=1635:00 +CPU= 25.994
  00:27:37 STEP 1636 H=1636:00 +CPU= 16.860
  00:27:54 STEP 1637 H=1637:00 +CPU= 16.348
  00:28:21 STEP 1638 H=1638:00 +CPU= 26.562
  00:28:37 STEP 1639 H=1639:00 +CPU= 16.359
  00:28:54 STEP 1640 H=1640:00 +CPU= 16.868
  00:29:21 STEP 1641 H=1641:00 +CPU= 26.015
  00:29:38 STEP 1642 H=1642:00 +CPU= 16.882
  00:29:55 STEP 1643 H=1643:00 +CPU= 16.394
  00:30:24 STEP 1644 H=1644:00 +CPU= 28.791
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001644
  00:30:41 STEP 1645 H=1645:00 +CPU= 16.414
  00:30:58 STEP 1646 H=1646:00 +CPU= 16.871
  00:31:24 STEP 1647 H=1647:00 +CPU= 26.013
  00:31:41 STEP 1648 H=1648:00 +CPU= 16.866
  00:31:58 STEP 1649 H=1649:00 +CPU= 16.392
  00:32:25 STEP 1650 H=1650:00 +CPU= 26.570
  00:32:41 STEP 1651 H=1651:00 +CPU= 16.387
  00:32:59 STEP 1652 H=1652:00 +CPU= 16.879
  00:33:25 STEP 1653 H=1653:00 +CPU= 26.007
  00:33:42 STEP 1654 H=1654:00 +CPU= 16.842
  00:33:59 STEP 1655 H=1655:00 +CPU= 16.841
  00:34:30 STEP 1656 H=1656:00 +CPU= 30.470
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189_68.zip
Uploading the intermediate file: upload_file_68.zip
  00:34:47 STEP 1657 H=1657:00 +CPU= 16.343
  00:35:04 STEP 1658 H=1658:00 +CPU= 16.840
  00:35:31 STEP 1659 H=1659:00 +CPU= 26.071
  00:35:48 STEP 1660 H=1660:00 +CPU= 16.876
  00:36:04 STEP 1661 H=1661:00 +CPU= 16.382
  00:36:31 STEP 1662 H=1662:00 +CPU= 26.610
  00:36:48 STEP 1663 H=1663:00 +CPU= 16.405
  00:37:05 STEP 1664 H=1664:00 +CPU= 16.934
  00:37:32 STEP 1665 H=1665:00 +CPU= 26.123
  00:37:49 STEP 1666 H=1666:00 +CPU= 16.921
  00:38:06 STEP 1667 H=1667:00 +CPU= 16.427
  00:38:35 STEP 1668 H=1668:00 +CPU= 28.743
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001668
  00:38:53 STEP 1669 H=1669:00 +CPU= 17.228
  00:39:15 STEP 1670 H=1670:00 +CPU= 18.363
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2002050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0440
(argv4) batchid: 971
(argv5) wuid: 12188084
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12188084
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c349c1784683faa42d2f0134579d5c74 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0440_2002050100_123_971_12188084.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0440_2002050100_123_971_12188084.zip
ic_ancil_file: ic_ancil_12188084
ifsdata_file: ifsdata_12188084
climate_data_file: clim_data_12188084
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_494768d76891e3f9dcfef535df354b75 to: /var/lib/boinc-client/slots/6/ic_ancil_12188084.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12188084.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_603411aaa83bb6c83c50f77facc271a0 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12188084.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12188084.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12188084.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12188084.zip
Checking for progress XML file: /var/lib/boinc-client/slots/6/progress_file_12188084.xml
Opened progress file ok : /var/lib/boinc-client/slots/6/progress_file_12188084.xml
-- Model is restarting --
Adjusting last_iter, 1671, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/6/progress_file_12188084.xml
last_cpu_time: 44387
upload_file_number: 69
last_iter: 1655
last_upload: 5961600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0440_2002050100_123_971_12188084_0_r1965900189
The child process has been launched with process id: 2148285
Executing the command: /var/lib/boinc-client/slots/6/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:pres-incite:1:1:2148285:2148285] [20230105:011614:1672899374:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:16:51 STEP 1656 H=1656:00 +CPU= 31.590
  01:17:15 STEP 1657 H=1657:00 +CPU= 17.195
  01:17:32 STEP 1658 H=1658:00 +CPU= 16.564
  01:17:58 STEP 1659 H=1659:00 +CPU= 25.731
  01:18:16 STEP 1660 H=1660:00 +CPU= 17.495
  01:18:34 STEP 1661 H=1661:00 +CPU= 17.630
  01:19:08 STEP 1662 H=1662:00 +CPU= 27.589
  01:19:24 STEP 1663 H=1663:00 +CPU= 16.141
  01:19:41 STEP 1664 H=1664:00 +CPU= 16.923
  01:20:09 STEP 1665 H=1665:00 +CPU= 26.683
  01:20:27 STEP 1666 H=1666:00 +CPU= 17.846
  01:20:53 STEP 1667 H=1667:00 +CPU= 17.497
  01:21:22 STEP 1668 H=1668:00 +CPU= 28.357
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhq0f+001668
  01:21:38 STEP 1669 H=1669:00 +CPU= 16.086
  01:21:55 STEP 1670 H=1670:00 +CPU= 16.765
  01:22:22 STEP 1671 H=1671:00 +CPU= 26.268
  01:22:39 STEP 1672 H=1672:00 +CPU= 16.962
  01:22:56 STEP 1673 H=1673:00 +CPU= 16.431
  01:23:23 STEP 1674 H=1674:00 +CPU= 26.797
  01:23:40 STEP 1675 H=1675:00 +CPU= 16.437
  01:23:57 STEP 1676 H=1676:00 +CPU= 16.925
  01:24:24 STEP 1677 H=1677:00 +CPU= 26.230
  01:24:41 STEP 1678 H=1678:00 +CPU= 16.950
  01:24:58 STEP 1679 H=1679:00 +CPU= 16.923
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:08:26 1528008 22273553 oifs_43r3_ps_0440_2002050100_123_971_12188084_0 4,492,800 35,195 0.0078
09 May 2023 16:08:16 1528008 22273553 oifs_43r3_ps_0440_2002050100_123_971_12188084_0 2,160,000 21,434 0.0099


©2024 climateprediction.net