climateprediction.net home page
Task 22445831

Task 22445831

Name oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0
Workunit 12292227
Created 13 Jun 2024, 1:40:32 UTC
Sent 13 Jun 2024, 1:54:23 UTC
Report deadline 12 Aug 2024, 1:54:23 UTC
Received 14 Jun 2024, 23:56:43 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1472385
Run time 16 hours 56 min 32 sec
CPU time 16 hours 56 min 32 sec
Validate state Valid
Credit 2,635.55
Device peak FLOPS 3.64 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,013.73 MB
Peak swap size 5,981.06 MB
Peak disk usage 1,582.12 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
it@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:14523:14523] [20240614:210016:1718388016:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:01:29 STEP 1440 H= 360:00 +CPU= 30.228
  21:02:00 STEP 1441 H= 360:15 +CPU= 30.502
  21:02:32 STEP 1442 H= 360:30 +CPU= 32.256
  21:03:03 STEP 1443 H= 360:45 +CPU= 29.444
  21:03:34 STEP 1444 H= 361:00 +CPU= 30.882
  21:04:05 STEP 1445 H= 361:15 +CPU= 29.913
  21:04:35 STEP 1446 H= 361:30 +CPU= 30.004
  21:05:02 STEP 1447 H= 361:45 +CPU= 25.752
  21:05:30 STEP 1448 H= 362:00 +CPU= 28.575
  21:05:57 STEP 1449 H= 362:15 +CPU= 26.410
  21:06:27 STEP 1450 H= 362:30 +CPU= 29.386
  21:06:58 STEP 1451 H= 362:45 +CPU= 30.998
  21:07:41 STEP 1452 H= 363:00 +CPU= 41.780
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001452
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001452
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001452
  21:08:14 STEP 1453 H= 363:15 +CPU= 31.490
  21:08:46 STEP 1454 H= 363:30 +CPU= 31.615
  21:09:17 STEP 1455 H= 363:45 +CPU= 31.224
  21:09:48 STEP 1456 H= 364:00 +CPU= 31.153
  21:10:21 STEP 1457 H= 364:15 +CPU= 30.965
  21:10:51 STEP 1458 H= 364:30 +CPU= 29.640
  21:11:19 STEP 1459 H= 364:45 +CPU= 28.462
  21:12:12 STEP 1460 H= 365:00 +CPU= 32.849
  21:12:44 STEP 1461 H= 365:15 +CPU= 32.528
  21:13:18 STEP 1462 H= 365:30 +CPU= 33.048
  21:13:46 STEP 1463 H= 365:45 +CPU= 28.062
  21:14:24 STEP 1464 H= 366:00 +CPU= 37.949
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001464
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001464
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001464
  21:14:55 STEP 1465 H= 366:15 +CPU= 30.955
  21:15:28 STEP 1466 H= 366:30 +CPU= 32.145
  21:15:58 STEP 1467 H= 366:45 +CPU= 30.124
  21:16:30 STEP 1468 H= 367:00 +CPU= 31.982
  21:17:00 STEP 1469 H= 367:15 +CPU= 29.932
  21:17:32 STEP 1470 H= 367:30 +CPU= 32.298
  21:18:04 STEP 1471 H= 367:45 +CPU= 31.171
  21:18:36 STEP 1472 H= 368:00 +CPU= 32.563
  21:19:06 STEP 1473 H= 368:15 +CPU= 29.082
  21:19:36 STEP 1474 H= 368:30 +CPU= 30.717
  21:20:07 STEP 1475 H= 368:45 +CPU= 30.500
  21:20:47 STEP 1476 H= 369:00 +CPU= 39.290
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001476
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001476
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001476
  21:21:15 STEP 1477 H= 369:15 +CPU= 28.508
  21:21:46 STEP 1478 H= 369:30 +CPU= 30.614
  21:22:16 STEP 1479 H= 369:45 +CPU= 30.143
  21:22:49 STEP 1480 H= 370:00 +CPU= 32.394
  21:23:20 STEP 1481 H= 370:15 +CPU= 31.226
  21:23:51 STEP 1482 H= 370:30 +CPU= 30.638
  21:24:22 STEP 1483 H= 370:45 +CPU= 30.438
  21:24:53 STEP 1484 H= 371:00 +CPU= 31.474
  21:25:25 STEP 1485 H= 371:15 +CPU= 31.739
  21:25:54 STEP 1486 H= 371:30 +CPU= 28.366
  21:26:23 STEP 1487 H= 371:45 +CPU= 29.237
  21:27:04 STEP 1488 H= 372:00 +CPU= 38.794
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001488
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001488
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001488
  21:27:32 STEP 1489 H= 372:15 +CPU= 27.963
  21:28:02 STEP 1490 H= 372:30 +CPU= 29.932
  21:28:35 STEP 1491 H= 372:45 +CPU= 32.381
  21:29:03 STEP 1492 H= 373:00 +CPU= 27.649
  21:29:31 STEP 1493 H= 373:15 +CPU= 28.412
  21:30:03 STEP 1494 H= 373:30 +CPU= 31.013
  21:30:32 STEP 1495 H= 373:45 +CPU= 29.355
  21:31:02 STEP 1496 H= 374:00 +CPU= 29.726
  21:31:36 STEP 1497 H= 374:15 +CPU= 34.070
  21:32:07 STEP 1498 H= 374:30 +CPU= 30.510
  21:32:39 STEP 1499 H= 374:45 +CPU= 31.515
  21:33:21 STEP 1500 H= 375:00 +CPU= 40.942
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001500
  21:33:53 STEP 1501 H= 375:15 +CPU= 32.092
  21:34:25 STEP 1502 H= 375:30 +CPU= 31.083
  21:34:55 STEP 1503 H= 375:45 +CPU= 30.135
  21:35:28 STEP 1504 H= 376:00 +CPU= 32.312
  21:35:58 STEP 1505 H= 376:15 +CPU= 29.387
  21:36:31 STEP 1506 H= 376:30 +CPU= 32.417
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a30c
(argv4) batchid: 1018
(argv5) wuid: 12292227
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/1
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_3795d6f84a2ce6cfcb17b2746593a3d8 to: /var/lib/boinc/slots/1/oifs_43r3_bl_a30c_2016092300_20_1018_12292227.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/1/oifs_43r3_bl_a30c_2016092300_20_1018_12292227.zip
ic_ancil_file: ic_ancil_12292227
ifsdata_file: ifsdata_12292227
climate_data_file: clim_data_12292227
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/1/ic_ancil_12292227.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/1/ic_ancil_12292227.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_925ffe003128eb2fb24ae737cc3fed92 to: /var/lib/boinc/slots/1/ifsdata/ifsdata_12292227.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/1/ifsdata/ifsdata_12292227.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/1/159l_2/clim_data_12292227.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/1/159l_2/clim_data_12292227.zip
Checking for progress XML file: /var/lib/boinc/slots/1/progress_file_12292227.xml
Opened progress file ok : /var/lib/boinc/slots/1/progress_file_12292227.xml
-- Model is restarting --
Adjusting last_iter, 1506, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/1/progress_file_12292227.xml
last_cpu_time: 48126
upload_file_number: 15
last_iter: 1489
last_upload: 1296000
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0_r2057787496
The child process has been launched with process id: 21230
Executing the command: /var/lib/boinc/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:kone6.localdomain:1:1:21230:21230] [20240614:221135:1718392295:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  22:12:18 STEP 1488 H= 372:00 +CPU= 28.918
  22:12:45 STEP 1489 H= 372:15 +CPU= 26.708
  22:13:12 STEP 1490 H= 372:30 +CPU= 26.616
  22:13:42 STEP 1491 H= 372:45 +CPU= 28.999
  22:14:09 STEP 1492 H= 373:00 +CPU= 26.623
  22:14:37 STEP 1493 H= 373:15 +CPU= 28.446
  22:15:06 STEP 1494 H= 373:30 +CPU= 28.328
  22:15:36 STEP 1495 H= 373:45 +CPU= 29.047
  22:16:02 STEP 1496 H= 374:00 +CPU= 26.094
  22:16:30 STEP 1497 H= 374:15 +CPU= 27.264
  22:16:56 STEP 1498 H= 374:30 +CPU= 25.742
  22:17:25 STEP 1499 H= 374:45 +CPU= 28.588
  22:18:02 STEP 1500 H= 375:00 +CPU= 36.696
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001500
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001500
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001500
  22:18:31 STEP 1501 H= 375:15 +CPU= 28.552
  22:19:00 STEP 1502 H= 375:30 +CPU= 28.536
  22:19:28 STEP 1503 H= 375:45 +CPU= 27.597
  22:19:57 STEP 1504 H= 376:00 +CPU= 28.467
  22:20:25 STEP 1505 H= 376:15 +CPU= 27.333
  22:20:54 STEP 1506 H= 376:30 +CPU= 29.090
  22:21:23 STEP 1507 H= 376:45 +CPU= 27.944
  22:21:52 STEP 1508 H= 377:00 +CPU= 29.233
  22:22:21 STEP 1509 H= 377:15 +CPU= 28.035
  22:22:51 STEP 1510 H= 377:30 +CPU= 29.293
  22:23:21 STEP 1511 H= 377:45 +CPU= 29.376
  22:23:56 STEP 1512 H= 378:00 +CPU= 34.834
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001512
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001512
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001512
  22:24:25 STEP 1513 H= 378:15 +CPU= 28.447
  22:24:52 STEP 1514 H= 378:30 +CPU= 27.284
  22:25:22 STEP 1515 H= 378:45 +CPU= 28.864
  22:25:52 STEP 1516 H= 379:00 +CPU= 29.831
  22:26:20 STEP 1517 H= 379:15 +CPU= 27.590
  22:26:49 STEP 1518 H= 379:30 +CPU= 28.480
  22:27:18 STEP 1519 H= 379:45 +CPU= 27.914
  22:27:48 STEP 1520 H= 380:00 +CPU= 29.830
  22:28:17 STEP 1521 H= 380:15 +CPU= 28.348
  22:28:45 STEP 1522 H= 380:30 +CPU= 27.785
  22:29:15 STEP 1523 H= 380:45 +CPU= 28.687
  22:29:51 STEP 1524 H= 381:00 +CPU= 35.509
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001524
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001524
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001524
  22:30:21 STEP 1525 H= 381:15 +CPU= 28.427
  22:30:49 STEP 1526 H= 381:30 +CPU= 26.589
  22:31:18 STEP 1527 H= 381:45 +CPU= 28.479
  22:31:49 STEP 1528 H= 382:00 +CPU= 29.709
  22:32:19 STEP 1529 H= 382:15 +CPU= 27.982
  22:32:48 STEP 1530 H= 382:30 +CPU= 28.941
  22:33:19 STEP 1531 H= 382:45 +CPU= 28.622
  22:33:49 STEP 1532 H= 383:00 +CPU= 29.383
  22:34:19 STEP 1533 H= 383:15 +CPU= 28.305
  22:34:47 STEP 1534 H= 383:30 +CPU= 28.079
  22:35:17 STEP 1535 H= 383:45 +CPU= 28.466
  22:35:58 STEP 1536 H= 384:00 +CPU= 39.762
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001536
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001536
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001452
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001476
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001488
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001500
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001512
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001524
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001524
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0_r2057787496_15.zip
Uploading the intermediate file: upload_file_15.zip
  22:36:29 STEP 1537 H= 384:15 +CPU= 28.670
  22:36:57 STEP 1538 H= 384:30 +CPU= 27.525
  22:37:27 STEP 1539 H= 384:45 +CPU= 29.007
  22:37:57 STEP 1540 H= 385:00 +CPU= 28.625
  22:38:26 STEP 1541 H= 385:15 +CPU= 28.740
  22:38:56 STEP 1542 H= 385:30 +CPU= 29.158
  22:39:25 STEP 1543 H= 385:45 +CPU= 29.057
  22:39:56 STEP 1544 H= 386:00 +CPU= 29.850
  22:40:25 STEP 1545 H= 386:15 +CPU= 28.598
  22:40:54 STEP 1546 H= 386:30 +CPU= 28.402
  22:41:24 STEP 1547 H= 386:45 +CPU= 29.658
  22:42:00 STEP 1548 H= 387:00 +CPU= 34.199
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001548
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001548
  22:42:29 STEP 1549 H= 387:15 +CPU= 29.005
  22:42:56 STEP 1550 H= 387:30 +CPU= 26.679
  22:43:25 STEP 1551 H= 387:45 +CPU= 28.084
  22:43:57 STEP 1552 H= 388:00 +CPU= 30.777
  22:44:26 STEP 1553 H= 388:15 +CPU= 28.924
  22:44:54 STEP 1554 H= 388:30 +CPU= 27.071
  22:45:25 STEP 1555 H= 388:45 +CPU= 30.043
  22:45:55 STEP 1556 H= 389:00 +CPU= 30.005
  22:46:25 STEP 1557 H= 389:15 +CPU= 29.090
  22:46:56 STEP 1558 H= 389:30 +CPU= 30.330
  22:47:26 STEP 1559 H= 389:45 +CPU= 29.237
  22:48:03 STEP 1560 H= 390:00 +CPU= 36.453
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001560
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001560
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001560
  22:48:32 STEP 1561 H= 390:15 +CPU= 28.845
  22:49:01 STEP 1562 H= 390:30 +CPU= 28.689
  22:49:33 STEP 1563 H= 390:45 +CPU= 30.269
  22:50:03 STEP 1564 H= 391:00 +CPU= 29.264
  22:50:33 STEP 1565 H= 391:15 +CPU= 29.207
  22:51:05 STEP 1566 H= 391:30 +CPU= 30.266
  22:51:35 STEP 1567 H= 391:45 +CPU= 29.348
  22:52:06 STEP 1568 H= 392:00 +CPU= 30.648
  22:52:35 STEP 1569 H= 392:15 +CPU= 28.612
  22:53:04 STEP 1570 H= 392:30 +CPU= 28.346
  22:53:33 STEP 1571 H= 392:45 +CPU= 28.004
  22:54:12 STEP 1572 H= 393:00 +CPU= 36.627
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001572
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001572
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001572
  22:54:44 STEP 1573 H= 393:15 +CPU= 30.128
  22:55:13 STEP 1574 H= 393:30 +CPU= 28.464
  22:55:43 STEP 1575 H= 393:45 +CPU= 29.590
  22:56:14 STEP 1576 H= 394:00 +CPU= 30.713
  22:56:42 STEP 1577 H= 394:15 +CPU= 27.809
  22:57:13 STEP 1578 H= 394:30 +CPU= 29.050
  22:57:41 STEP 1579 H= 394:45 +CPU= 28.053
  22:58:11 STEP 1580 H= 395:00 +CPU= 29.261
  22:58:40 STEP 1581 H= 395:15 +CPU= 28.323
  22:59:09 STEP 1582 H= 395:30 +CPU= 27.614
  22:59:40 STEP 1583 H= 395:45 +CPU= 29.883
  23:00:18 STEP 1584 H= 396:00 +CPU= 37.549
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001584
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001584
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001584
  23:00:47 STEP 1585 H= 396:15 +CPU= 28.782
  23:01:17 STEP 1586 H= 396:30 +CPU= 28.649
  23:01:46 STEP 1587 H= 396:45 +CPU= 28.133
  23:02:17 STEP 1588 H= 397:00 +CPU= 29.522
  23:02:44 STEP 1589 H= 397:15 +CPU= 26.505
  23:03:10 STEP 1590 H= 397:30 +CPU= 26.462
  23:03:41 STEP 1591 H= 397:45 +CPU= 29.811
  23:04:10 STEP 1592 H= 398:00 +CPU= 28.633
  23:04:41 STEP 1593 H= 398:15 +CPU= 30.567
  23:05:11 STEP 1594 H= 398:30 +CPU= 28.615
  23:05:42 STEP 1595 H= 398:45 +CPU= 29.604
  23:06:21 STEP 1596 H= 399:00 +CPU= 36.334
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001596
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001596
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001596
  23:06:50 STEP 1597 H= 399:15 +CPU= 28.045
  23:07:20 STEP 1598 H= 399:30 +CPU= 29.230
  23:07:50 STEP 1599 H= 399:45 +CPU= 29.768
  23:08:19 STEP 1600 H= 400:00 +CPU= 28.301
  23:08:48 STEP 1601 H= 400:15 +CPU= 28.825
  23:09:17 STEP 1602 H= 400:30 +CPU= 28.541
  23:09:46 STEP 1603 H= 400:45 +CPU= 28.308
  23:10:19 STEP 1604 H= 401:00 +CPU= 30.357
  23:10:51 STEP 1605 H= 401:15 +CPU= 29.325
  23:11:19 STEP 1606 H= 401:30 +CPU= 28.602
  23:11:50 STEP 1607 H= 401:45 +CPU= 29.981
  23:12:25 STEP 1608 H= 402:00 +CPU= 35.487
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001608
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001608
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001608
  23:12:54 STEP 1609 H= 402:15 +CPU= 28.553
  23:13:25 STEP 1610 H= 402:30 +CPU= 29.984
  23:13:58 STEP 1611 H= 402:45 +CPU= 30.450
  23:14:26 STEP 1612 H= 403:00 +CPU= 27.549
  23:14:56 STEP 1613 H= 403:15 +CPU= 29.989
  23:15:26 STEP 1614 H= 403:30 +CPU= 28.971
  23:15:55 STEP 1615 H= 403:45 +CPU= 27.962
  23:16:25 STEP 1616 H= 404:00 +CPU= 29.940
  23:16:54 STEP 1617 H= 404:15 +CPU= 28.352
  23:17:28 STEP 1618 H= 404:30 +CPU= 30.664
  23:17:58 STEP 1619 H= 404:45 +CPU= 29.103
  23:18:36 STEP 1620 H= 405:00 +CPU= 37.631
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001620
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001620
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001620
  23:19:07 STEP 1621 H= 405:15 +CPU= 30.490
  23:19:35 STEP 1622 H= 405:30 +CPU= 27.667
  23:20:05 STEP 1623 H= 405:45 +CPU= 29.305
  23:20:36 STEP 1624 H= 406:00 +CPU= 30.771
  23:21:07 STEP 1625 H= 406:15 +CPU= 29.477
  23:21:35 STEP 1626 H= 406:30 +CPU= 28.218
  23:22:04 STEP 1627 H= 406:45 +CPU= 28.037
  23:22:35 STEP 1628 H= 407:00 +CPU= 30.593
  23:23:04 STEP 1629 H= 407:15 +CPU= 28.796
  23:23:36 STEP 1630 H= 407:30 +CPU= 29.775
  23:24:06 STEP 1631 H= 407:45 +CPU= 28.624
  23:24:46 STEP 1632 H= 408:00 +CPU= 39.955
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001632
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001632
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001536
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001548
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001560
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001572
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001584
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001596
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001608
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001620
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001620
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0_r2057787496_16.zip
Uploading the intermediate file: upload_file_16.zip
  23:25:15 STEP 1633 H= 408:15 +CPU= 28.504
  23:25:47 STEP 1634 H= 408:30 +CPU= 29.476
  23:26:17 STEP 1635 H= 408:45 +CPU= 29.313
  23:26:49 STEP 1636 H= 409:00 +CPU= 30.749
  23:27:18 STEP 1637 H= 409:15 +CPU= 29.006
  23:27:50 STEP 1638 H= 409:30 +CPU= 29.568
  23:28:19 STEP 1639 H= 409:45 +CPU= 28.492
  23:28:47 STEP 1640 H= 410:00 +CPU= 27.958
  23:29:18 STEP 1641 H= 410:15 +CPU= 29.727
  23:29:53 STEP 1642 H= 410:30 +CPU= 30.415
  23:30:23 STEP 1643 H= 410:45 +CPU= 29.543
  23:30:59 STEP 1644 H= 411:00 +CPU= 35.682
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001644
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001644
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001644
  23:31:29 STEP 1645 H= 411:15 +CPU= 29.115
  23:32:02 STEP 1646 H= 411:30 +CPU= 30.654
  23:32:32 STEP 1647 H= 411:45 +CPU= 28.286
  23:33:02 STEP 1648 H= 412:00 +CPU= 29.414
  23:33:32 STEP 1649 H= 412:15 +CPU= 29.030
  23:34:02 STEP 1650 H= 412:30 +CPU= 29.318
  23:34:30 STEP 1651 H= 412:45 +CPU= 28.282
  23:35:01 STEP 1652 H= 413:00 +CPU= 29.564
  23:35:30 STEP 1653 H= 413:15 +CPU= 28.954
  23:36:01 STEP 1654 H= 413:30 +CPU= 30.033
  23:36:31 STEP 1655 H= 413:45 +CPU= 28.441
  23:37:09 STEP 1656 H= 414:00 +CPU= 37.168
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001656
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001656
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001656
  23:37:40 STEP 1657 H= 414:15 +CPU= 30.320
  23:38:09 STEP 1658 H= 414:30 +CPU= 28.062
  23:38:38 STEP 1659 H= 414:45 +CPU= 29.081
  23:39:09 STEP 1660 H= 415:00 +CPU= 31.348
  23:39:38 STEP 1661 H= 415:15 +CPU= 28.525
  23:40:10 STEP 1662 H= 415:30 +CPU= 29.986
  23:40:40 STEP 1663 H= 415:45 +CPU= 28.588
  23:41:11 STEP 1664 H= 416:00 +CPU= 30.104
  23:41:40 STEP 1665 H= 416:15 +CPU= 28.781
  23:42:10 STEP 1666 H= 416:30 +CPU= 30.094
  23:42:41 STEP 1667 H= 416:45 +CPU= 29.570
  23:43:22 STEP 1668 H= 417:00 +CPU= 37.386
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001668
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001668
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001668
  23:43:53 STEP 1669 H= 417:15 +CPU= 29.920
  23:44:23 STEP 1670 H= 417:30 +CPU= 29.277
  23:44:53 STEP 1671 H= 417:45 +CPU= 29.793
  23:45:23 STEP 1672 H= 418:00 +CPU= 28.900
  23:45:52 STEP 1673 H= 418:15 +CPU= 29.020
  23:46:23 STEP 1674 H= 418:30 +CPU= 28.964
  23:46:56 STEP 1675 H= 418:45 +CPU= 30.176
  23:47:27 STEP 1676 H= 419:00 +CPU= 29.633
  23:47:56 STEP 1677 H= 419:15 +CPU= 28.696
  23:48:25 STEP 1678 H= 419:30 +CPU= 29.084
  23:48:53 STEP 1679 H= 419:45 +CPU= 27.705
  23:49:34 STEP 1680 H= 420:00 +CPU= 39.524
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001680
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001680
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001680
  23:50:06 STEP 1681 H= 420:15 +CPU= 30.223
  23:50:34 STEP 1682 H= 420:30 +CPU= 27.457
  23:51:02 STEP 1683 H= 420:45 +CPU= 27.291
  23:51:32 STEP 1684 H= 421:00 +CPU= 29.553
  23:52:00 STEP 1685 H= 421:15 +CPU= 28.064
  23:52:31 STEP 1686 H= 421:30 +CPU= 29.858
  23:53:03 STEP 1687 H= 421:45 +CPU= 29.463
  23:53:33 STEP 1688 H= 422:00 +CPU= 28.649
  23:54:01 STEP 1689 H= 422:15 +CPU= 27.738
  23:54:29 STEP 1690 H= 422:30 +CPU= 27.792
  23:54:59 STEP 1691 H= 422:45 +CPU= 29.140
  23:55:38 STEP 1692 H= 423:00 +CPU= 37.247
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001692
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001692
  23:56:06 STEP 1693 H= 423:15 +CPU= 27.017
  23:56:35 STEP 1694 H= 423:30 +CPU= 28.821
  23:57:07 STEP 1695 H= 423:45 +CPU= 30.772
  23:57:39 STEP 1696 H= 424:00 +CPU= 30.972
  23:58:07 STEP 1697 H= 424:15 +CPU= 28.230
  23:58:35 STEP 1698 H= 424:30 +CPU= 27.253
  23:59:05 STEP 1699 H= 424:45 +CPU= 29.522
  23:59:35 STEP 1700 H= 425:00 +CPU= 29.905
  00:00:05 STEP 1701 H= 425:15 +CPU= 29.133
  00:00:36 STEP 1702 H= 425:30 +CPU= 29.030
  00:01:05 STEP 1703 H= 425:45 +CPU= 28.859
  00:01:42 STEP 1704 H= 426:00 +CPU= 36.348
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001704
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001704
  00:02:10 STEP 1705 H= 426:15 +CPU= 27.130
  00:02:40 STEP 1706 H= 426:30 +CPU= 29.274
  00:03:09 STEP 1707 H= 426:45 +CPU= 29.047
  00:03:39 STEP 1708 H= 427:00 +CPU= 29.472
  00:04:07 STEP 1709 H= 427:15 +CPU= 27.996
  00:04:38 STEP 1710 H= 427:30 +CPU= 29.751
  00:05:08 STEP 1711 H= 427:45 +CPU= 29.089
  00:05:40 STEP 1712 H= 428:00 +CPU= 30.942
  00:06:10 STEP 1713 H= 428:15 +CPU= 29.259
  00:06:39 STEP 1714 H= 428:30 +CPU= 28.984
  00:07:09 STEP 1715 H= 428:45 +CPU= 29.039
  00:07:48 STEP 1716 H= 429:00 +CPU= 37.958
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001716
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001716
  00:08:17 STEP 1717 H= 429:15 +CPU= 28.965
  00:08:47 STEP 1718 H= 429:30 +CPU= 28.971
  00:09:15 STEP 1719 H= 429:45 +CPU= 27.860
  00:09:44 STEP 1720 H= 430:00 +CPU= 28.228
  00:10:16 STEP 1721 H= 430:15 +CPU= 29.789
  00:10:44 STEP 1722 H= 430:30 +CPU= 28.139
  00:11:13 STEP 1723 H= 430:45 +CPU= 28.140
  00:11:42 STEP 1724 H= 431:00 +CPU= 28.842
  00:12:11 STEP 1725 H= 431:15 +CPU= 28.679
  00:12:40 STEP 1726 H= 431:30 +CPU= 28.092
  00:13:10 STEP 1727 H= 431:45 +CPU= 29.993
  00:13:52 STEP 1728 H= 432:00 +CPU= 40.730
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001728
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001632
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001644
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001656
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001668
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0_r2057787496_17.zip
Uploading the intermediate file: upload_file_17.zip
  00:14:23 STEP 1729 H= 432:15 +CPU= 29.492
  00:14:52 STEP 1730 H= 432:30 +CPU= 28.604
  00:15:21 STEP 1731 H= 432:45 +CPU= 27.852
  00:15:52 STEP 1732 H= 433:00 +CPU= 30.361
  00:16:22 STEP 1733 H= 433:15 +CPU= 29.951
  00:16:51 STEP 1734 H= 433:30 +CPU= 28.128
  00:17:21 STEP 1735 H= 433:45 +CPU= 28.842
  00:17:52 STEP 1736 H= 434:00 +CPU= 30.982
  00:18:22 STEP 1737 H= 434:15 +CPU= 28.653
  00:18:50 STEP 1738 H= 434:30 +CPU= 28.223
  00:19:20 STEP 1739 H= 434:45 +CPU= 29.026
  00:19:56 STEP 1740 H= 435:00 +CPU= 35.837
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001740
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001740
  00:20:27 STEP 1741 H= 435:15 +CPU= 29.314
  00:20:56 STEP 1742 H= 435:30 +CPU= 28.833
  00:21:27 STEP 1743 H= 435:45 +CPU= 30.097
  00:21:57 STEP 1744 H= 436:00 +CPU= 29.260
  00:22:27 STEP 1745 H= 436:15 +CPU= 29.369
  00:22:55 STEP 1746 H= 436:30 +CPU= 27.611
  00:23:25 STEP 1747 H= 436:45 +CPU= 29.110
  00:23:54 STEP 1748 H= 437:00 +CPU= 28.531
  00:24:23 STEP 1749 H= 437:15 +CPU= 28.163
  00:24:51 STEP 1750 H= 437:30 +CPU= 28.519
  00:25:22 STEP 1751 H= 437:45 +CPU= 29.152
  00:25:57 STEP 1752 H= 438:00 +CPU= 35.168
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001752
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001752
  00:26:28 STEP 1753 H= 438:15 +CPU= 29.491
  00:26:56 STEP 1754 H= 438:30 +CPU= 26.854
  00:27:25 STEP 1755 H= 438:45 +CPU= 28.280
  00:27:55 STEP 1756 H= 439:00 +CPU= 30.244
  00:28:25 STEP 1757 H= 439:15 +CPU= 29.276
  00:28:55 STEP 1758 H= 439:30 +CPU= 29.281
  00:29:24 STEP 1759 H= 439:45 +CPU= 28.354
  00:29:55 STEP 1760 H= 440:00 +CPU= 30.498
  00:30:24 STEP 1761 H= 440:15 +CPU= 28.648
  00:30:53 STEP 1762 H= 440:30 +CPU= 28.010
  00:31:22 STEP 1763 H= 440:45 +CPU= 28.664
  00:31:58 STEP 1764 H= 441:00 +CPU= 35.866
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001764
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001764
  00:32:29 STEP 1765 H= 441:15 +CPU= 29.751
  00:32:59 STEP 1766 H= 441:30 +CPU= 29.141
  00:33:28 STEP 1767 H= 441:45 +CPU= 28.889
  00:33:58 STEP 1768 H= 442:00 +CPU= 29.386
  00:34:27 STEP 1769 H= 442:15 +CPU= 28.309
  00:34:57 STEP 1770 H= 442:30 +CPU= 29.610
  00:35:27 STEP 1771 H= 442:45 +CPU= 29.293
  00:35:57 STEP 1772 H= 443:00 +CPU= 29.194
  00:36:25 STEP 1773 H= 443:15 +CPU= 28.270
  00:36:56 STEP 1774 H= 443:30 +CPU= 29.909
  00:37:25 STEP 1775 H= 443:45 +CPU= 28.882
  00:38:05 STEP 1776 H= 444:00 +CPU= 39.526
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001776
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001776
  00:38:34 STEP 1777 H= 444:15 +CPU= 28.487
  00:39:04 STEP 1778 H= 444:30 +CPU= 29.450
  00:39:34 STEP 1779 H= 444:45 +CPU= 29.792
  00:40:05 STEP 1780 H= 445:00 +CPU= 30.100
  00:40:36 STEP 1781 H= 445:15 +CPU= 30.205
  00:41:04 STEP 1782 H= 445:30 +CPU= 28.060
  00:41:34 STEP 1783 H= 445:45 +CPU= 28.722
  00:42:03 STEP 1784 H= 446:00 +CPU= 29.048
  00:42:33 STEP 1785 H= 446:15 +CPU= 28.854
  00:43:03 STEP 1786 H= 446:30 +CPU= 29.347
  00:43:31 STEP 1787 H= 446:45 +CPU= 27.968
  00:44:07 STEP 1788 H= 447:00 +CPU= 35.568
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001788
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001788
  00:44:38 STEP 1789 H= 447:15 +CPU= 29.602
  00:45:07 STEP 1790 H= 447:30 +CPU= 28.937
  00:45:35 STEP 1791 H= 447:45 +CPU= 27.581
  00:46:05 STEP 1792 H= 448:00 +CPU= 29.408
  00:46:35 STEP 1793 H= 448:15 +CPU= 28.964
  00:47:04 STEP 1794 H= 448:30 +CPU= 28.634
  00:47:32 STEP 1795 H= 448:45 +CPU= 28.079
  00:48:03 STEP 1796 H= 449:00 +CPU= 29.890
  00:48:33 STEP 1797 H= 449:15 +CPU= 29.396
  00:49:03 STEP 1798 H= 449:30 +CPU= 29.042
  00:49:32 STEP 1799 H= 449:45 +CPU= 28.573
  00:50:09 STEP 1800 H= 450:00 +CPU= 36.863
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001800
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001800
  00:50:39 STEP 1801 H= 450:15 +CPU= 28.777
  00:51:06 STEP 1802 H= 450:30 +CPU= 27.255
  00:51:34 STEP 1803 H= 450:45 +CPU= 27.651
  00:52:04 STEP 1804 H= 451:00 +CPU= 28.891
  00:52:32 STEP 1805 H= 451:15 +CPU= 27.115
  00:53:02 STEP 1806 H= 451:30 +CPU= 29.667
  00:53:32 STEP 1807 H= 451:45 +CPU= 29.536
  00:54:04 STEP 1808 H= 452:00 +CPU= 30.813
  00:54:34 STEP 1809 H= 452:15 +CPU= 29.933
  00:55:04 STEP 1810 H= 452:30 +CPU= 29.811
  00:55:33 STEP 1811 H= 452:45 +CPU= 27.988
  00:56:09 STEP 1812 H= 453:00 +CPU= 36.107
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001812
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001812
  00:56:38 STEP 1813 H= 453:15 +CPU= 28.729
  00:57:07 STEP 1814 H= 453:30 +CPU= 27.993
  00:57:37 STEP 1815 H= 453:45 +CPU= 29.481
  00:58:07 STEP 1816 H= 454:00 +CPU= 29.033
  00:58:36 STEP 1817 H= 454:15 +CPU= 29.108
  00:59:04 STEP 1818 H= 454:30 +CPU= 27.455
  00:59:35 STEP 1819 H= 454:45 +CPU= 29.834
  01:00:05 STEP 1820 H= 455:00 +CPU= 29.141
  01:00:34 STEP 1821 H= 455:15 +CPU= 28.680
  01:01:04 STEP 1822 H= 455:30 +CPU= 29.272
  01:01:34 STEP 1823 H= 455:45 +CPU= 28.896
  01:02:16 STEP 1824 H= 456:00 +CPU= 40.164
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001824
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0_r2057787496_18.zip
Uploading the intermediate file: upload_file_18.zip
  01:02:46 STEP 1825 H= 456:15 +CPU= 28.797
  01:03:16 STEP 1826 H= 456:30 +CPU= 29.570
  01:03:48 STEP 1827 H= 456:45 +CPU= 30.926
  01:04:18 STEP 1828 H= 457:00 +CPU= 29.320
  01:04:49 STEP 1829 H= 457:15 +CPU= 29.873
  01:05:19 STEP 1830 H= 457:30 +CPU= 29.779
  01:05:49 STEP 1831 H= 457:45 +CPU= 29.306
  01:06:20 STEP 1832 H= 458:00 +CPU= 30.994
  01:06:50 STEP 1833 H= 458:15 +CPU= 29.045
  01:07:20 STEP 1834 H= 458:30 +CPU= 29.485
  01:07:51 STEP 1835 H= 458:45 +CPU= 30.130
  01:08:28 STEP 1836 H= 459:00 +CPU= 36.406
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001836
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001836
  01:08:58 STEP 1837 H= 459:15 +CPU= 29.046
  01:09:29 STEP 1838 H= 459:30 +CPU= 30.108
  01:09:58 STEP 1839 H= 459:45 +CPU= 28.510
  01:10:30 STEP 1840 H= 460:00 +CPU= 29.741
  01:11:00 STEP 1841 H= 460:15 +CPU= 29.129
  01:11:32 STEP 1842 H= 460:30 +CPU= 30.507
  01:12:05 STEP 1843 H= 460:45 +CPU= 29.778
  01:12:37 STEP 1844 H= 461:00 +CPU= 31.141
  01:13:09 STEP 1845 H= 461:15 +CPU= 30.000
  01:13:42 STEP 1846 H= 461:30 +CPU= 30.078
  01:14:16 STEP 1847 H= 461:45 +CPU= 30.278
  01:14:56 STEP 1848 H= 462:00 +CPU= 36.703
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001848
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001848
  01:15:28 STEP 1849 H= 462:15 +CPU= 29.526
  01:16:05 STEP 1850 H= 462:30 +CPU= 30.284
  01:16:38 STEP 1851 H= 462:45 +CPU= 30.331
  01:17:15 STEP 1852 H= 463:00 +CPU= 30.802
  01:17:51 STEP 1853 H= 463:15 +CPU= 29.799
  01:18:26 STEP 1854 H= 463:30 +CPU= 29.604
  01:19:03 STEP 1855 H= 463:45 +CPU= 30.173
  01:19:37 STEP 1856 H= 464:00 +CPU= 31.584
  01:20:11 STEP 1857 H= 464:15 +CPU= 30.044
  01:20:46 STEP 1858 H= 464:30 +CPU= 29.833
  01:21:18 STEP 1859 H= 464:45 +CPU= 30.883
  01:22:01 STEP 1860 H= 465:00 +CPU= 36.553
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001860
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001860
  01:22:35 STEP 1861 H= 465:15 +CPU= 29.753
  01:23:07 STEP 1862 H= 465:30 +CPU= 29.123
  01:23:45 STEP 1863 H= 465:45 +CPU= 29.826
  01:24:20 STEP 1864 H= 466:00 +CPU= 31.634
  01:24:59 STEP 1865 H= 466:15 +CPU= 30.126
  01:25:32 STEP 1866 H= 466:30 +CPU= 30.031
  01:26:04 STEP 1867 H= 466:45 +CPU= 30.276
  01:26:45 STEP 1868 H= 467:00 +CPU= 31.125
  01:27:18 STEP 1869 H= 467:15 +CPU= 29.624
  01:27:54 STEP 1870 H= 467:30 +CPU= 30.818
  01:28:27 STEP 1871 H= 467:45 +CPU= 29.743
  01:29:17 STEP 1872 H= 468:00 +CPU= 41.310
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001872
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001872
  01:29:50 STEP 1873 H= 468:15 +CPU= 30.324
  01:30:24 STEP 1874 H= 468:30 +CPU= 30.103
  01:31:01 STEP 1875 H= 468:45 +CPU= 30.574
  01:31:37 STEP 1876 H= 469:00 +CPU= 31.447
  01:32:14 STEP 1877 H= 469:15 +CPU= 29.439
  01:32:49 STEP 1878 H= 469:30 +CPU= 30.983
  01:33:22 STEP 1879 H= 469:45 +CPU= 29.693
  01:33:55 STEP 1880 H= 470:00 +CPU= 31.261
  01:34:27 STEP 1881 H= 470:15 +CPU= 30.447
  01:34:59 STEP 1882 H= 470:30 +CPU= 30.496
  01:35:33 STEP 1883 H= 470:45 +CPU= 29.920
  01:36:14 STEP 1884 H= 471:00 +CPU= 37.929
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001884
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001884
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001884
  01:36:47 STEP 1885 H= 471:15 +CPU= 30.071
  01:37:23 STEP 1886 H= 471:30 +CPU= 30.066
  01:37:58 STEP 1887 H= 471:45 +CPU= 30.173
  01:38:32 STEP 1888 H= 472:00 +CPU= 30.647
  01:39:12 STEP 1889 H= 472:15 +CPU= 30.015
  01:39:44 STEP 1890 H= 472:30 +CPU= 29.999
  01:40:18 STEP 1891 H= 472:45 +CPU= 29.463
  01:40:47 STEP 1892 H= 473:00 +CPU= 28.990
  01:41:14 STEP 1893 H= 473:15 +CPU= 27.127
  01:41:43 STEP 1894 H= 473:30 +CPU= 27.545
  01:42:12 STEP 1895 H= 473:45 +CPU= 27.974
  01:42:44 STEP 1896 H= 474:00 +CPU= 32.435
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001896
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001896
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001896
  01:43:12 STEP 1897 H= 474:15 +CPU= 27.390
  01:43:40 STEP 1898 H= 474:30 +CPU= 28.049
  01:44:09 STEP 1899 H= 474:45 +CPU= 28.567
  01:44:37 STEP 1900 H= 475:00 +CPU= 27.786
  01:45:06 STEP 1901 H= 475:15 +CPU= 28.282
  01:45:33 STEP 1902 H= 475:30 +CPU= 26.705
  01:46:01 STEP 1903 H= 475:45 +CPU= 27.717
  01:46:32 STEP 1904 H= 476:00 +CPU= 30.299
  01:47:01 STEP 1905 H= 476:15 +CPU= 28.626
  01:47:29 STEP 1906 H= 476:30 +CPU= 27.864
  01:47:57 STEP 1907 H= 476:45 +CPU= 27.289
  01:48:33 STEP 1908 H= 477:00 +CPU= 36.098
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001908
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001908
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001908
  01:49:01 STEP 1909 H= 477:15 +CPU= 27.415
  01:49:30 STEP 1910 H= 477:30 +CPU= 29.319
  01:49:59 STEP 1911 H= 477:45 +CPU= 27.938
  01:50:29 STEP 1912 H= 478:00 +CPU= 29.791
  01:50:59 STEP 1913 H= 478:15 +CPU= 29.813
  01:51:27 STEP 1914 H= 478:30 +CPU= 27.363
  01:51:56 STEP 1915 H= 478:45 +CPU= 28.919
  01:52:22 STEP 1916 H= 479:00 +CPU= 26.016
  01:52:48 STEP 1917 H= 479:15 +CPU= 25.261
  01:53:17 STEP 1918 H= 479:30 +CPU= 28.471
  01:53:47 STEP 1919 H= 479:45 +CPU= 29.033
  01:54:25 STEP 1920 H= 480:00 +CPU= 37.604
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  480.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.1294871277E-20      I       0.3860337507E-07      I
ICloud fractionI      0.0000000000E+00      I       0.1102107150E-01      I
I  Relat. Hum. I      0.3804697759E+00      I       0.3987334313E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.4601774871E+02      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1920 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001920 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001920
 MPI-TASK:   1 -    298208196 BYTES IN      7 RECORDS TRANSFERRED IN    0.0002 SECONDS******** Mbytes/s, TOTAL TIME=    0.0077( 2.6%)
 01:54:25 STEP 1920 H= 480:00 +CPU= 37.604
 END CNT3
 NSTEP =  1920 CNT0   000000000  27.768  27.768  28.267   0.982               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN        12870/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    228    0       46      1044068    12870 ************    31.0    11.8    19.2    12.4    43.4
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 ********* *********    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 ********* *********     99.90      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  469    4657.7    4657.7     16.34      0.00
   9 SPCM     - SPECTRAL COMP.                     432     850.1     850.1      2.75      0.00
  10 SCAN2M - PHYSICS                              433   15056.9   15056.9     48.76      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 36     756.9     756.9      0.20      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 19      46.8      46.8      0.01      0.00
  17 GRIDFPOS IN CNT4                               36      33.9      33.9      0.01      0.00
  27 RESTART FILES - WRITING                         9    3027.7    3027.7      0.20      0.00
  28 RESTART FILES - READING                         1   13014.9   13014.9      0.10      0.00
  29 SU4FPOS IN CNT4                                36       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                36    6635.5    6635.5      1.79      0.00
  31 POSDDH IN STEPO                                 2      76.6      76.6      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    433    6407.0    6407.0     20.75      0.00
  39 SU0YOMB                                         1     285.7     285.7      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    433      30.3      30.3      0.10      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              432     289.9     289.9      0.94      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              432      57.2      57.2      0.18      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 19       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      940     465.9     465.9      3.28      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       900     281.0     281.0      1.89      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        900      99.1      99.1      0.67      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       940     155.1     155.1      1.09      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      46.2      46.2      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            940      51.5      51.5      0.36      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            900      27.4      27.4      0.18      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            940      97.3      97.3      0.68      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            900      24.5      24.5      0.16      0.00
 400 GSTATS                                      51056       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 47450       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME    13369.9 CPU TIME   12866.8 VECTOR TIME    12866.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      129.4
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/1/ICMGGh7zg+001920
Moving to projects directory: /var/lib/boinc/slots/1/ICMSHh7zg+001920
Moving to projects directory: /var/lib/boinc/slots/1/ICMUAh7zg+001920
Adding to the zip: /var/lib/boinc/slots/1/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/1/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMGGh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMSHh7zg+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12292227/ICMUAh7zg+001920
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0_r2057787496_19.zip
Uploading the final file: upload_file_19.zip
Uploading trickle at timestep: 1727100
01:57:57 (21226): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Jun 2024 22:56:07 1472385 22445831 oifs_43r3_bl_a30c_2016092300_20_1018_12292227_0 1,727,100 60,992 0.0353


©2024 cpdn.org