climateprediction.net home page
Task 22251466

Task 22251466

Name oifs_43r3_ps_0606_1981050100_123_950_12167250_0
Workunit 12167250
Created 20 Dec 2022, 19:57:49 UTC
Sent 20 Dec 2022, 20:00:48 UTC
Report deadline 19 Jan 2023, 20:00:48 UTC
Received 23 Dec 2022, 4:28:43 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1494992
Run time 16 hours 47 min 54 sec
CPU time 16 hours 23 min 46 sec
Validate state Invalid
Credit 10,039.68
Device peak FLOPS 5.32 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,419.07 MB
Peak swap size 4,981.06 MB
Peak disk usage 7,688.32 MB
Stderr 
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
1699:00 +CPU= 21.552
  21:48:30 STEP 1700 H=1700:00 +CPU= 22.000
  21:49:03 STEP 1701 H=1701:00 +CPU= 33.342
  21:49:26 STEP 1702 H=1702:00 +CPU= 22.257
  21:49:48 STEP 1703 H=1703:00 +CPU= 21.972
  21:50:28 STEP 1704 H=1704:00 +CPU= 39.669
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_70.zip
Uploading the intermediate file: upload_file_70.zip
  21:50:50 STEP 1705 H=1705:00 +CPU= 20.996
  21:51:12 STEP 1706 H=1706:00 +CPU= 22.129
  21:51:47 STEP 1707 H=1707:00 +CPU= 34.429
  21:52:10 STEP 1708 H=1708:00 +CPU= 22.994
  21:52:32 STEP 1709 H=1709:00 +CPU= 21.682
  21:53:08 STEP 1710 H=1710:00 +CPU= 35.203
  21:53:30 STEP 1711 H=1711:00 +CPU= 21.983
  21:53:52 STEP 1712 H=1712:00 +CPU= 22.243
  21:54:26 STEP 1713 H=1713:00 +CPU= 33.642
  21:54:48 STEP 1714 H=1714:00 +CPU= 21.986
  21:55:10 STEP 1715 H=1715:00 +CPU= 21.111
  21:55:47 STEP 1716 H=1716:00 +CPU= 36.992
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001716
  21:56:09 STEP 1717 H=1717:00 +CPU= 21.317
  21:56:31 STEP 1718 H=1718:00 +CPU= 22.334
  21:57:05 STEP 1719 H=1719:00 +CPU= 33.147
  21:57:27 STEP 1720 H=1720:00 +CPU= 21.976
  21:57:49 STEP 1721 H=1721:00 +CPU= 21.558
  21:58:24 STEP 1722 H=1722:00 +CPU= 34.215
  21:58:46 STEP 1723 H=1723:00 +CPU= 21.498
  21:59:09 STEP 1724 H=1724:00 +CPU= 22.668
  21:59:44 STEP 1725 H=1725:00 +CPU= 34.297
  22:00:07 STEP 1726 H=1726:00 +CPU= 22.490
  22:00:30 STEP 1727 H=1727:00 +CPU= 22.841
  22:01:11 STEP 1728 H=1728:00 +CPU= 40.552
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_71.zip
Uploading the intermediate file: upload_file_71.zip
  22:01:33 STEP 1729 H=1729:00 +CPU= 21.160
  22:01:55 STEP 1730 H=1730:00 +CPU= 22.289
  22:02:29 STEP 1731 H=1731:00 +CPU= 33.490
  22:02:52 STEP 1732 H=1732:00 +CPU= 22.914
  22:03:14 STEP 1733 H=1733:00 +CPU= 21.812
  22:03:49 STEP 1734 H=1734:00 +CPU= 35.016
  22:04:11 STEP 1735 H=1735:00 +CPU= 21.421
  22:04:33 STEP 1736 H=1736:00 +CPU= 22.181
  22:05:07 STEP 1737 H=1737:00 +CPU= 33.286
  22:05:29 STEP 1738 H=1738:00 +CPU= 21.854
  22:05:51 STEP 1739 H=1739:00 +CPU= 21.696
  22:06:28 STEP 1740 H=1740:00 +CPU= 36.715
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001740
  22:06:50 STEP 1741 H=1741:00 +CPU= 21.869
  22:07:13 STEP 1742 H=1742:00 +CPU= 22.933
  22:07:49 STEP 1743 H=1743:00 +CPU= 35.137
  22:08:13 STEP 1744 H=1744:00 +CPU= 23.173
  22:08:35 STEP 1745 H=1745:00 +CPU= 22.057
  22:09:10 STEP 1746 H=1746:00 +CPU= 34.483
  22:09:31 STEP 1747 H=1747:00 +CPU= 21.178
  22:09:54 STEP 1748 H=1748:00 +CPU= 22.095
  22:10:27 STEP 1749 H=1749:00 +CPU= 33.418
  22:10:50 STEP 1750 H=1750:00 +CPU= 22.091
  22:11:12 STEP 1751 H=1751:00 +CPU= 22.206
  22:11:54 STEP 1752 H=1752:00 +CPU= 40.174
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_72.zip
Uploading the intermediate file: upload_file_72.zip
  22:12:16 STEP 1753 H=1753:00 +CPU= 21.243
  22:12:39 STEP 1754 H=1754:00 +CPU= 22.327
  22:13:13 STEP 1755 H=1755:00 +CPU= 33.380
  22:13:36 STEP 1756 H=1756:00 +CPU= 22.525
  22:13:58 STEP 1757 H=1757:00 +CPU= 21.372
  22:14:33 STEP 1758 H=1758:00 +CPU= 35.047
  22:14:56 STEP 1759 H=1759:00 +CPU= 22.143
  22:15:19 STEP 1760 H=1760:00 +CPU= 22.836
  22:15:54 STEP 1761 H=1761:00 +CPU= 34.712
  22:16:16 STEP 1762 H=1762:00 +CPU= 22.597
  22:16:38 STEP 1763 H=1763:00 +CPU= 21.548
  22:17:16 STEP 1764 H=1764:00 +CPU= 36.917
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001764
  22:17:37 STEP 1765 H=1765:00 +CPU= 21.388
  22:18:00 STEP 1766 H=1766:00 +CPU= 22.326
  22:18:34 STEP 1767 H=1767:00 +CPU= 33.290
  22:18:57 STEP 1768 H=1768:00 +CPU= 22.277
  22:19:18 STEP 1769 H=1769:00 +CPU= 21.700
  22:19:53 STEP 1770 H=1770:00 +CPU= 34.436
  22:20:14 STEP 1771 H=1771:00 +CPU= 20.924
  22:20:36 STEP 1772 H=1772:00 +CPU= 21.302
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0606
(argv4) batchid: 950
(argv5) wuid: 12167250
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/11
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e1805dc70f64b91beb5dbf54ccd0ab0a to: /var/lib/boinc/slots/11/oifs_43r3_ps_0606_1981050100_123_950_12167250.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_0606_1981050100_123_950_12167250.zip
ic_ancil_file: ic_ancil_12167250
ifsdata_file: ifsdata_12167250
climate_data_file: clim_data_12167250
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2eb859624378ecde760e5cc436b014ff to: /var/lib/boinc/slots/11/ic_ancil_12167250.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/11/ic_ancil_12167250.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eff821229d255c5719c79573ce17c94e to: /var/lib/boinc/slots/11/ifsdata/ifsdata_12167250.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/11/ifsdata/ifsdata_12167250.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/11/159l_2/clim_data_12167250.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/11/159l_2/clim_data_12167250.zip
Checking for progress XML file: /var/lib/boinc/slots/11/progress_file_12167250.xml
Opened progress file ok : /var/lib/boinc/slots/11/progress_file_12167250.xml
-- Model is restarting --
Adjusting last_iter, 1773, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/11/progress_file_12167250.xml
last_cpu_time: 48272
upload_file_number: 73
last_iter: 1751
last_upload: 6307200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543
The child process has been launched with process id: 60023
Executing the command: /var/lib/boinc/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:60023:60023] [20221223:012615:1671755175:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:26:58 STEP 1752 H=1752:00 +CPU= 38.439
  01:27:20 STEP 1753 H=1753:00 +CPU= 21.725
  01:27:42 STEP 1754 H=1754:00 +CPU= 21.987
  01:28:16 STEP 1755 H=1755:00 +CPU= 33.451
  01:28:39 STEP 1756 H=1756:00 +CPU= 22.388
  01:29:00 STEP 1757 H=1757:00 +CPU= 21.407
  01:29:35 STEP 1758 H=1758:00 +CPU= 34.508
  01:29:56 STEP 1759 H=1759:00 +CPU= 21.313
  01:30:19 STEP 1760 H=1760:00 +CPU= 22.212
  01:30:53 STEP 1761 H=1761:00 +CPU= 33.066
  01:31:15 STEP 1762 H=1762:00 +CPU= 22.267
  01:31:36 STEP 1763 H=1763:00 +CPU= 21.227
  01:32:15 STEP 1764 H=1764:00 +CPU= 38.094
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001764
  01:32:37 STEP 1765 H=1765:00 +CPU= 22.182
  01:33:01 STEP 1766 H=1766:00 +CPU= 23.024
  01:33:36 STEP 1767 H=1767:00 +CPU= 34.438
  01:33:59 STEP 1768 H=1768:00 +CPU= 22.685
  01:34:20 STEP 1769 H=1769:00 +CPU= 21.506
  01:34:55 STEP 1770 H=1770:00 +CPU= 33.824
  01:35:17 STEP 1771 H=1771:00 +CPU= 21.400
  01:35:39 STEP 1772 H=1772:00 +CPU= 22.092
  01:36:14 STEP 1773 H=1773:00 +CPU= 33.785
  01:36:36 STEP 1774 H=1774:00 +CPU= 22.046
  01:36:58 STEP 1775 H=1775:00 +CPU= 22.081
  01:37:39 STEP 1776 H=1776:00 +CPU= 39.818
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_73.zip
Uploading the intermediate file: upload_file_73.zip
  01:38:01 STEP 1777 H=1777:00 +CPU= 21.468
  01:38:23 STEP 1778 H=1778:00 +CPU= 22.250
  01:38:56 STEP 1779 H=1779:00 +CPU= 32.920
  01:39:19 STEP 1780 H=1780:00 +CPU= 22.438
  01:39:42 STEP 1781 H=1781:00 +CPU= 22.857
  01:40:18 STEP 1782 H=1782:00 +CPU= 35.427
  01:40:41 STEP 1783 H=1783:00 +CPU= 21.870
  01:41:05 STEP 1784 H=1784:00 +CPU= 22.890
  01:41:39 STEP 1785 H=1785:00 +CPU= 34.178
  01:42:02 STEP 1786 H=1786:00 +CPU= 22.501
  01:42:24 STEP 1787 H=1787:00 +CPU= 21.465
  01:43:01 STEP 1788 H=1788:00 +CPU= 36.855
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001788
  01:43:25 STEP 1789 H=1789:00 +CPU= 22.376
  01:43:49 STEP 1790 H=1790:00 +CPU= 22.746
  01:44:22 STEP 1791 H=1791:00 +CPU= 33.310
  01:44:45 STEP 1792 H=1792:00 +CPU= 21.975
  01:45:06 STEP 1793 H=1793:00 +CPU= 21.336
  01:45:41 STEP 1794 H=1794:00 +CPU= 34.214
  01:46:02 STEP 1795 H=1795:00 +CPU= 21.474
  01:46:25 STEP 1796 H=1796:00 +CPU= 22.371
  01:46:59 STEP 1797 H=1797:00 +CPU= 33.686
  01:47:21 STEP 1798 H=1798:00 +CPU= 22.060
  01:47:43 STEP 1799 H=1799:00 +CPU= 22.018
  01:48:25 STEP 1800 H=1800:00 +CPU= 40.292
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_74.zip
Uploading the intermediate file: upload_file_74.zip
  01:48:47 STEP 1801 H=1801:00 +CPU= 21.487
  01:49:10 STEP 1802 H=1802:00 +CPU= 22.664
  01:49:44 STEP 1803 H=1803:00 +CPU= 33.756
  01:50:07 STEP 1804 H=1804:00 +CPU= 22.366
  01:50:28 STEP 1805 H=1805:00 +CPU= 21.646
  01:51:04 STEP 1806 H=1806:00 +CPU= 35.247
  01:51:26 STEP 1807 H=1807:00 +CPU= 22.050
  01:51:48 STEP 1808 H=1808:00 +CPU= 22.291
  01:52:23 STEP 1809 H=1809:00 +CPU= 34.166
  01:52:45 STEP 1810 H=1810:00 +CPU= 22.255
  01:53:07 STEP 1811 H=1811:00 +CPU= 21.536
  01:53:45 STEP 1812 H=1812:00 +CPU= 37.721
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001812
  01:54:07 STEP 1813 H=1813:00 +CPU= 22.078
  01:54:30 STEP 1814 H=1814:00 +CPU= 22.454
  01:55:04 STEP 1815 H=1815:00 +CPU= 33.696
  01:55:27 STEP 1816 H=1816:00 +CPU= 22.552
  01:55:48 STEP 1817 H=1817:00 +CPU= 21.645
  01:56:23 STEP 1818 H=1818:00 +CPU= 34.733
  01:56:45 STEP 1819 H=1819:00 +CPU= 21.795
  01:57:08 STEP 1820 H=1820:00 +CPU= 22.492
  01:57:41 STEP 1821 H=1821:00 +CPU= 32.987
  01:58:04 STEP 1822 H=1822:00 +CPU= 22.665
  01:58:27 STEP 1823 H=1823:00 +CPU= 22.535
  01:59:08 STEP 1824 H=1824:00 +CPU= 40.158
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_75.zip
Uploading the intermediate file: upload_file_75.zip
  01:59:29 STEP 1825 H=1825:00 +CPU= 21.330
  01:59:51 STEP 1826 H=1826:00 +CPU= 21.819
  02:00:25 STEP 1827 H=1827:00 +CPU= 33.556
  02:00:47 STEP 1828 H=1828:00 +CPU= 21.917
  02:01:09 STEP 1829 H=1829:00 +CPU= 21.580
  02:01:44 STEP 1830 H=1830:00 +CPU= 34.439
  02:02:05 STEP 1831 H=1831:00 +CPU= 21.572
  02:02:28 STEP 1832 H=1832:00 +CPU= 22.675
  02:03:03 STEP 1833 H=1833:00 +CPU= 34.415
  02:03:25 STEP 1834 H=1834:00 +CPU= 22.347
  02:03:47 STEP 1835 H=1835:00 +CPU= 21.542
  02:04:25 STEP 1836 H=1836:00 +CPU= 37.578
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001836
  02:04:47 STEP 1837 H=1837:00 +CPU= 21.683
  02:05:10 STEP 1838 H=1838:00 +CPU= 22.503
  02:05:43 STEP 1839 H=1839:00 +CPU= 33.315
  02:06:06 STEP 1840 H=1840:00 +CPU= 22.983
  02:06:28 STEP 1841 H=1841:00 +CPU= 21.462
  02:07:03 STEP 1842 H=1842:00 +CPU= 34.358
  02:07:25 STEP 1843 H=1843:00 +CPU= 21.603
  02:07:47 STEP 1844 H=1844:00 +CPU= 22.643
  02:08:22 STEP 1845 H=1845:00 +CPU= 34.357
  02:08:45 STEP 1846 H=1846:00 +CPU= 22.312
  02:09:07 STEP 1847 H=1847:00 +CPU= 22.012
  02:09:47 STEP 1848 H=1848:00 +CPU= 39.920
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_76.zip
Uploading the intermediate file: upload_file_76.zip
  02:10:09 STEP 1849 H=1849:00 +CPU= 21.537
  02:10:32 STEP 1850 H=1850:00 +CPU= 22.143
  02:11:05 STEP 1851 H=1851:00 +CPU= 33.257
  02:11:28 STEP 1852 H=1852:00 +CPU= 21.983
  02:11:49 STEP 1853 H=1853:00 +CPU= 21.503
  02:12:24 STEP 1854 H=1854:00 +CPU= 34.425
  02:12:46 STEP 1855 H=1855:00 +CPU= 21.328
  02:13:08 STEP 1856 H=1856:00 +CPU= 22.656
  02:13:43 STEP 1857 H=1857:00 +CPU= 34.164
  02:14:05 STEP 1858 H=1858:00 +CPU= 22.016
  02:14:27 STEP 1859 H=1859:00 +CPU= 21.996
  02:15:06 STEP 1860 H=1860:00 +CPU= 38.262
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001860
  02:15:28 STEP 1861 H=1861:00 +CPU= 22.318
  02:15:51 STEP 1862 H=1862:00 +CPU= 22.499
  02:16:25 STEP 1863 H=1863:00 +CPU= 33.764
  02:16:48 STEP 1864 H=1864:00 +CPU= 22.544
  02:17:10 STEP 1865 H=1865:00 +CPU= 21.279
  02:17:44 STEP 1866 H=1866:00 +CPU= 34.049
  02:18:06 STEP 1867 H=1867:00 +CPU= 21.672
  02:18:29 STEP 1868 H=1868:00 +CPU= 22.533
  02:19:03 STEP 1869 H=1869:00 +CPU= 33.703
  02:19:25 STEP 1870 H=1870:00 +CPU= 22.309
  02:19:48 STEP 1871 H=1871:00 +CPU= 22.545
  02:20:29 STEP 1872 H=1872:00 +CPU= 39.769
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_77.zip
Uploading the intermediate file: upload_file_77.zip
  02:20:50 STEP 1873 H=1873:00 +CPU= 21.402
  02:21:12 STEP 1874 H=1874:00 +CPU= 21.816
  02:21:46 STEP 1875 H=1875:00 +CPU= 33.319
  02:22:09 STEP 1876 H=1876:00 +CPU= 22.743
  02:22:31 STEP 1877 H=1877:00 +CPU= 21.998
  02:23:06 STEP 1878 H=1878:00 +CPU= 34.445
  02:23:27 STEP 1879 H=1879:00 +CPU= 21.310
  02:23:50 STEP 1880 H=1880:00 +CPU= 22.806
  02:24:24 STEP 1881 H=1881:00 +CPU= 33.465
  02:24:47 STEP 1882 H=1882:00 +CPU= 22.179
  02:25:08 STEP 1883 H=1883:00 +CPU= 21.603
  02:25:46 STEP 1884 H=1884:00 +CPU= 37.241
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001884
  02:26:08 STEP 1885 H=1885:00 +CPU= 21.416
  02:26:29 STEP 1886 H=1886:00 +CPU= 21.592
  02:27:04 STEP 1887 H=1887:00 +CPU= 33.896
  02:27:26 STEP 1888 H=1888:00 +CPU= 22.461
  02:27:48 STEP 1889 H=1889:00 +CPU= 21.521
  02:28:23 STEP 1890 H=1890:00 +CPU= 34.131
  02:28:45 STEP 1891 H=1891:00 +CPU= 21.775
  02:29:07 STEP 1892 H=1892:00 +CPU= 22.096
  02:29:41 STEP 1893 H=1893:00 +CPU= 33.884
  02:30:04 STEP 1894 H=1894:00 +CPU= 22.172
  02:30:26 STEP 1895 H=1895:00 +CPU= 22.094
  02:31:06 STEP 1896 H=1896:00 +CPU= 39.735
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_78.zip
Uploading the intermediate file: upload_file_78.zip
  02:31:27 STEP 1897 H=1897:00 +CPU= 20.962
  02:31:49 STEP 1898 H=1898:00 +CPU= 21.695
  02:32:22 STEP 1899 H=1899:00 +CPU= 33.232
  02:32:45 STEP 1900 H=1900:00 +CPU= 22.626
  02:33:07 STEP 1901 H=1901:00 +CPU= 21.542
  02:33:42 STEP 1902 H=1902:00 +CPU= 34.788
  02:34:04 STEP 1903 H=1903:00 +CPU= 22.103
  02:34:27 STEP 1904 H=1904:00 +CPU= 22.625
  02:35:02 STEP 1905 H=1905:00 +CPU= 33.996
  02:35:24 STEP 1906 H=1906:00 +CPU= 22.470
  02:35:47 STEP 1907 H=1907:00 +CPU= 21.959
  02:36:25 STEP 1908 H=1908:00 +CPU= 37.696
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001908
  02:36:46 STEP 1909 H=1909:00 +CPU= 21.517
  02:37:09 STEP 1910 H=1910:00 +CPU= 22.379
  02:37:43 STEP 1911 H=1911:00 +CPU= 33.827
  02:38:06 STEP 1912 H=1912:00 +CPU= 22.291
  02:38:28 STEP 1913 H=1913:00 +CPU= 21.847
  02:39:03 STEP 1914 H=1914:00 +CPU= 34.941
  02:39:25 STEP 1915 H=1915:00 +CPU= 21.637
  02:39:47 STEP 1916 H=1916:00 +CPU= 22.252
  02:40:22 STEP 1917 H=1917:00 +CPU= 34.080
  02:40:44 STEP 1918 H=1918:00 +CPU= 22.681
  02:41:07 STEP 1919 H=1919:00 +CPU= 22.518
  02:41:48 STEP 1920 H=1920:00 +CPU= 40.708
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_79.zip
Uploading the intermediate file: upload_file_79.zip
  02:42:10 STEP 1921 H=1921:00 +CPU= 21.611
  02:42:32 STEP 1922 H=1922:00 +CPU= 22.083
  02:43:06 STEP 1923 H=1923:00 +CPU= 33.170
  02:43:29 STEP 1924 H=1924:00 +CPU= 22.609
  02:43:51 STEP 1925 H=1925:00 +CPU= 21.893
  02:44:26 STEP 1926 H=1926:00 +CPU= 34.867
  02:44:48 STEP 1927 H=1927:00 +CPU= 21.792
  02:45:10 STEP 1928 H=1928:00 +CPU= 21.984
  02:45:44 STEP 1929 H=1929:00 +CPU= 33.600
  02:46:07 STEP 1930 H=1930:00 +CPU= 22.518
  02:46:29 STEP 1931 H=1931:00 +CPU= 21.892
  02:47:06 STEP 1932 H=1932:00 +CPU= 37.646
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001932
  02:47:28 STEP 1933 H=1933:00 +CPU= 21.860
  02:47:51 STEP 1934 H=1934:00 +CPU= 22.169
  02:48:24 STEP 1935 H=1935:00 +CPU= 33.229
  02:48:47 STEP 1936 H=1936:00 +CPU= 22.491
  02:49:09 STEP 1937 H=1937:00 +CPU= 21.687
  02:49:43 STEP 1938 H=1938:00 +CPU= 33.911
  02:50:05 STEP 1939 H=1939:00 +CPU= 21.859
  02:50:28 STEP 1940 H=1940:00 +CPU= 22.690
  02:51:01 STEP 1941 H=1941:00 +CPU= 33.296
  02:51:24 STEP 1942 H=1942:00 +CPU= 22.170
  02:51:46 STEP 1943 H=1943:00 +CPU= 21.913
  02:52:25 STEP 1944 H=1944:00 +CPU= 39.422
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_80.zip
Uploading the intermediate file: upload_file_80.zip
  02:52:47 STEP 1945 H=1945:00 +CPU= 21.427
  02:53:09 STEP 1946 H=1946:00 +CPU= 22.052
  02:53:44 STEP 1947 H=1947:00 +CPU= 34.356
  02:54:07 STEP 1948 H=1948:00 +CPU= 22.733
  02:54:29 STEP 1949 H=1949:00 +CPU= 21.561
  02:55:04 STEP 1950 H=1950:00 +CPU= 34.796
  02:55:26 STEP 1951 H=1951:00 +CPU= 21.755
  02:55:49 STEP 1952 H=1952:00 +CPU= 22.655
  02:56:23 STEP 1953 H=1953:00 +CPU= 33.625
  02:56:45 STEP 1954 H=1954:00 +CPU= 22.321
  02:57:07 STEP 1955 H=1955:00 +CPU= 21.783
  02:57:44 STEP 1956 H=1956:00 +CPU= 36.853
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001956
  02:58:06 STEP 1957 H=1957:00 +CPU= 21.550
  02:58:29 STEP 1958 H=1958:00 +CPU= 22.673
  02:59:04 STEP 1959 H=1959:00 +CPU= 34.034
  02:59:25 STEP 1960 H=1960:00 +CPU= 21.638
  02:59:47 STEP 1961 H=1961:00 +CPU= 21.678
  03:00:21 STEP 1962 H=1962:00 +CPU= 33.863
  03:00:43 STEP 1963 H=1963:00 +CPU= 21.681
  03:01:06 STEP 1964 H=1964:00 +CPU= 22.533
  03:01:40 STEP 1965 H=1965:00 +CPU= 33.797
  03:02:03 STEP 1966 H=1966:00 +CPU= 22.255
  03:02:25 STEP 1967 H=1967:00 +CPU= 22.605
  03:03:05 STEP 1968 H=1968:00 +CPU= 39.455
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001968
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001956
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001956
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_81.zip
Uploading the intermediate file: upload_file_81.zip
  03:03:27 STEP 1969 H=1969:00 +CPU= 21.547
  03:03:49 STEP 1970 H=1970:00 +CPU= 22.027
  03:04:23 STEP 1971 H=1971:00 +CPU= 33.230
  03:04:45 STEP 1972 H=1972:00 +CPU= 22.168
  03:05:06 STEP 1973 H=1973:00 +CPU= 20.792
  03:05:40 STEP 1974 H=1974:00 +CPU= 33.225
  03:06:01 STEP 1975 H=1975:00 +CPU= 21.178
  03:06:24 STEP 1976 H=1976:00 +CPU= 22.409
  03:06:58 STEP 1977 H=1977:00 +CPU= 33.696
  03:07:20 STEP 1978 H=1978:00 +CPU= 22.397
  03:07:42 STEP 1979 H=1979:00 +CPU= 21.750
  03:08:21 STEP 1980 H=1980:00 +CPU= 37.869
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001980
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001980
  03:08:42 STEP 1981 H=1981:00 +CPU= 21.520
  03:09:05 STEP 1982 H=1982:00 +CPU= 22.593
  03:09:39 STEP 1983 H=1983:00 +CPU= 34.043
  03:10:02 STEP 1984 H=1984:00 +CPU= 22.488
  03:10:24 STEP 1985 H=1985:00 +CPU= 21.334
  03:10:58 STEP 1986 H=1986:00 +CPU= 34.530
  03:11:21 STEP 1987 H=1987:00 +CPU= 21.937
  03:11:44 STEP 1988 H=1988:00 +CPU= 22.894
  03:12:18 STEP 1989 H=1989:00 +CPU= 34.056
  03:12:40 STEP 1990 H=1990:00 +CPU= 22.167
  03:13:03 STEP 1991 H=1991:00 +CPU= 22.608
  03:13:44 STEP 1992 H=1992:00 +CPU= 40.840
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+001992
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001968
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001980
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001980
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_82.zip
Uploading the intermediate file: upload_file_82.zip
  03:14:06 STEP 1993 H=1993:00 +CPU= 21.663
Uploading trickle at timestep: 7171200
  03:14:29 STEP 1994 H=1994:00 +CPU= 22.417
  03:15:03 STEP 1995 H=1995:00 +CPU= 33.864
  03:15:26 STEP 1996 H=1996:00 +CPU= 22.401
  03:15:48 STEP 1997 H=1997:00 +CPU= 21.672
  03:16:23 STEP 1998 H=1998:00 +CPU= 35.002
  03:16:45 STEP 1999 H=1999:00 +CPU= 22.026
  03:17:08 STEP 2000 H=2000:00 +CPU= 22.496
  03:17:43 STEP 2001 H=2001:00 +CPU= 34.366
  03:18:06 STEP 2002 H=2002:00 +CPU= 22.890
  03:18:29 STEP 2003 H=2003:00 +CPU= 22.966
  03:19:09 STEP 2004 H=2004:00 +CPU= 39.212
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002004
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002004
  03:19:31 STEP 2005 H=2005:00 +CPU= 21.788
  03:19:53 STEP 2006 H=2006:00 +CPU= 21.865
  03:20:27 STEP 2007 H=2007:00 +CPU= 34.024
  03:20:50 STEP 2008 H=2008:00 +CPU= 22.426
  03:21:11 STEP 2009 H=2009:00 +CPU= 21.169
  03:21:47 STEP 2010 H=2010:00 +CPU= 35.256
  03:22:10 STEP 2011 H=2011:00 +CPU= 22.339
  03:22:33 STEP 2012 H=2012:00 +CPU= 22.762
  03:23:07 STEP 2013 H=2013:00 +CPU= 34.184
  03:23:30 STEP 2014 H=2014:00 +CPU= 22.416
  03:23:53 STEP 2015 H=2015:00 +CPU= 22.589
  03:24:34 STEP 2016 H=2016:00 +CPU= 40.933
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002016
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+001992
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002004
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002004
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_83.zip
Uploading the intermediate file: upload_file_83.zip
  03:24:57 STEP 2017 H=2017:00 +CPU= 22.748
  03:25:21 STEP 2018 H=2018:00 +CPU= 23.860
  03:25:57 STEP 2019 H=2019:00 +CPU= 35.782
  03:26:22 STEP 2020 H=2020:00 +CPU= 24.098
  03:26:44 STEP 2021 H=2021:00 +CPU= 22.673
  03:27:20 STEP 2022 H=2022:00 +CPU= 34.709
  03:27:42 STEP 2023 H=2023:00 +CPU= 21.868
  03:28:04 STEP 2024 H=2024:00 +CPU= 22.124
  03:28:38 STEP 2025 H=2025:00 +CPU= 33.513
  03:29:01 STEP 2026 H=2026:00 +CPU= 23.048
  03:29:23 STEP 2027 H=2027:00 +CPU= 22.108
  03:30:01 STEP 2028 H=2028:00 +CPU= 37.483
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002028
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002028
  03:30:23 STEP 2029 H=2029:00 +CPU= 21.916
  03:30:46 STEP 2030 H=2030:00 +CPU= 21.929
  03:31:19 STEP 2031 H=2031:00 +CPU= 33.027
  03:31:42 STEP 2032 H=2032:00 +CPU= 22.426
  03:32:04 STEP 2033 H=2033:00 +CPU= 21.420
  03:32:39 STEP 2034 H=2034:00 +CPU= 35.190
  03:33:02 STEP 2035 H=2035:00 +CPU= 22.632
  03:33:25 STEP 2036 H=2036:00 +CPU= 23.244
  03:33:59 STEP 2037 H=2037:00 +CPU= 33.734
  03:34:22 STEP 2038 H=2038:00 +CPU= 22.362
  03:34:44 STEP 2039 H=2039:00 +CPU= 21.653
  03:35:25 STEP 2040 H=2040:00 +CPU= 40.804
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002040
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002016
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002028
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002028
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_84.zip
Uploading the intermediate file: upload_file_84.zip
  03:35:47 STEP 2041 H=2041:00 +CPU= 21.476
  03:36:10 STEP 2042 H=2042:00 +CPU= 23.167
  03:36:44 STEP 2043 H=2043:00 +CPU= 33.841
  03:37:07 STEP 2044 H=2044:00 +CPU= 22.279
  03:37:28 STEP 2045 H=2045:00 +CPU= 21.191
  03:38:04 STEP 2046 H=2046:00 +CPU= 35.655
  03:38:27 STEP 2047 H=2047:00 +CPU= 22.456
  03:38:49 STEP 2048 H=2048:00 +CPU= 22.601
  03:39:25 STEP 2049 H=2049:00 +CPU= 34.772
  03:39:48 STEP 2050 H=2050:00 +CPU= 23.181
  03:40:10 STEP 2051 H=2051:00 +CPU= 22.302
  03:40:50 STEP 2052 H=2052:00 +CPU= 38.894
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002052
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002052
  03:41:11 STEP 2053 H=2053:00 +CPU= 21.513
  03:41:34 STEP 2054 H=2054:00 +CPU= 22.490
  03:42:08 STEP 2055 H=2055:00 +CPU= 33.476
  03:42:30 STEP 2056 H=2056:00 +CPU= 21.997
  03:42:52 STEP 2057 H=2057:00 +CPU= 21.394
  03:43:27 STEP 2058 H=2058:00 +CPU= 34.738
  03:43:49 STEP 2059 H=2059:00 +CPU= 21.762
  03:44:11 STEP 2060 H=2060:00 +CPU= 21.988
  03:44:44 STEP 2061 H=2061:00 +CPU= 33.453
  03:45:07 STEP 2062 H=2062:00 +CPU= 21.884
  03:45:30 STEP 2063 H=2063:00 +CPU= 23.136
  03:46:11 STEP 2064 H=2064:00 +CPU= 40.297
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002064
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002040
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002052
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002052
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_85.zip
Uploading the intermediate file: upload_file_85.zip
  03:46:32 STEP 2065 H=2065:00 +CPU= 21.369
  03:46:55 STEP 2066 H=2066:00 +CPU= 22.419
  03:47:29 STEP 2067 H=2067:00 +CPU= 33.264
  03:47:51 STEP 2068 H=2068:00 +CPU= 21.930
  03:48:13 STEP 2069 H=2069:00 +CPU= 21.308
  03:48:47 STEP 2070 H=2070:00 +CPU= 34.561
  03:49:09 STEP 2071 H=2071:00 +CPU= 21.951
  03:49:32 STEP 2072 H=2072:00 +CPU= 22.590
  03:50:06 STEP 2073 H=2073:00 +CPU= 33.884
  03:50:29 STEP 2074 H=2074:00 +CPU= 22.513
  03:50:50 STEP 2075 H=2075:00 +CPU= 21.507
  03:51:28 STEP 2076 H=2076:00 +CPU= 37.383
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002076
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002076
  03:51:50 STEP 2077 H=2077:00 +CPU= 21.790
  03:52:13 STEP 2078 H=2078:00 +CPU= 22.484
  03:52:47 STEP 2079 H=2079:00 +CPU= 33.466
  03:53:11 STEP 2080 H=2080:00 +CPU= 23.566
  03:53:34 STEP 2081 H=2081:00 +CPU= 22.763
  03:54:09 STEP 2082 H=2082:00 +CPU= 35.120
  03:54:32 STEP 2083 H=2083:00 +CPU= 22.598
  03:54:55 STEP 2084 H=2084:00 +CPU= 22.502
  03:55:28 STEP 2085 H=2085:00 +CPU= 32.843
  03:55:50 STEP 2086 H=2086:00 +CPU= 21.876
  03:56:13 STEP 2087 H=2087:00 +CPU= 22.984
  03:56:53 STEP 2088 H=2088:00 +CPU= 40.353
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002088
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002064
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002076
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002076
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_86.zip
Uploading the intermediate file: upload_file_86.zip
  03:57:15 STEP 2089 H=2089:00 +CPU= 21.377
  03:57:37 STEP 2090 H=2090:00 +CPU= 22.272
  03:58:12 STEP 2091 H=2091:00 +CPU= 33.801
  03:58:34 STEP 2092 H=2092:00 +CPU= 22.401
  03:58:56 STEP 2093 H=2093:00 +CPU= 21.432
  03:59:30 STEP 2094 H=2094:00 +CPU= 34.256
  03:59:52 STEP 2095 H=2095:00 +CPU= 21.594
  04:00:15 STEP 2096 H=2096:00 +CPU= 23.107
  04:00:51 STEP 2097 H=2097:00 +CPU= 35.434
  04:01:15 STEP 2098 H=2098:00 +CPU= 23.860
  04:01:37 STEP 2099 H=2099:00 +CPU= 21.662
  04:02:14 STEP 2100 H=2100:00 +CPU= 37.092
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002100
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002100
  04:02:36 STEP 2101 H=2101:00 +CPU= 21.714
  04:02:59 STEP 2102 H=2102:00 +CPU= 22.489
  04:03:33 STEP 2103 H=2103:00 +CPU= 33.891
  04:03:56 STEP 2104 H=2104:00 +CPU= 22.593
  04:04:18 STEP 2105 H=2105:00 +CPU= 21.662
  04:04:53 STEP 2106 H=2106:00 +CPU= 34.309
  04:05:15 STEP 2107 H=2107:00 +CPU= 21.935
  04:05:38 STEP 2108 H=2108:00 +CPU= 22.533
  04:06:11 STEP 2109 H=2109:00 +CPU= 33.355
  04:06:33 STEP 2110 H=2110:00 +CPU= 21.819
  04:06:57 STEP 2111 H=2111:00 +CPU= 23.765
  04:07:40 STEP 2112 H=2112:00 +CPU= 41.575
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002112
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002088
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002100
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002100
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_87.zip
Uploading the intermediate file: upload_file_87.zip
  04:08:02 STEP 2113 H=2113:00 +CPU= 22.474
  04:08:26 STEP 2114 H=2114:00 +CPU= 23.082
  04:09:01 STEP 2115 H=2115:00 +CPU= 34.504
  04:09:23 STEP 2116 H=2116:00 +CPU= 22.413
  04:09:46 STEP 2117 H=2117:00 +CPU= 21.656
  04:10:20 STEP 2118 H=2118:00 +CPU= 34.389
  04:10:43 STEP 2119 H=2119:00 +CPU= 22.110
  04:11:06 STEP 2120 H=2120:00 +CPU= 22.606
  04:11:39 STEP 2121 H=2121:00 +CPU= 33.002
  04:12:01 STEP 2122 H=2122:00 +CPU= 21.828
  04:12:22 STEP 2123 H=2123:00 +CPU= 21.300
  04:13:01 STEP 2124 H=2124:00 +CPU= 37.748
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002124
  04:13:23 STEP 2125 H=2125:00 +CPU= 21.883
  04:13:47 STEP 2126 H=2126:00 +CPU= 23.744
  04:14:21 STEP 2127 H=2127:00 +CPU= 34.389
  04:14:45 STEP 2128 H=2128:00 +CPU= 23.260
  04:15:07 STEP 2129 H=2129:00 +CPU= 22.516
  04:15:44 STEP 2130 H=2130:00 +CPU= 36.465
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1981050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0606
(argv4) batchid: 950
(argv5) wuid: 12167250
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/11
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e1805dc70f64b91beb5dbf54ccd0ab0a to: /var/lib/boinc/slots/11/oifs_43r3_ps_0606_1981050100_123_950_12167250.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/11/oifs_43r3_ps_0606_1981050100_123_950_12167250.zip
ic_ancil_file: ic_ancil_12167250
ifsdata_file: ifsdata_12167250
climate_data_file: clim_data_12167250
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_2eb859624378ecde760e5cc436b014ff to: /var/lib/boinc/slots/11/ic_ancil_12167250.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/11/ic_ancil_12167250.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eff821229d255c5719c79573ce17c94e to: /var/lib/boinc/slots/11/ifsdata/ifsdata_12167250.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/11/ifsdata/ifsdata_12167250.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/11/159l_2/clim_data_12167250.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/11/159l_2/clim_data_12167250.zip
Checking for progress XML file: /var/lib/boinc/slots/11/progress_file_12167250.xml
Opened progress file ok : /var/lib/boinc/slots/11/progress_file_12167250.xml
-- Model is restarting --
Adjusting last_iter, 2131, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/11/progress_file_12167250.xml
last_cpu_time: 58336
upload_file_number: 88
last_iter: 2111
last_upload: 7603200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543
The child process has been launched with process id: 67578
Executing the command: /var/lib/boinc/slots/11/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:proxy-server:1:1:67578:67578] [20221223:043240:1671766360:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  04:33:19 STEP 2112 H=2112:00 +CPU= 36.383
  04:33:40 STEP 2113 H=2113:00 +CPU= 20.462
  04:34:02 STEP 2114 H=2114:00 +CPU= 21.280
  04:34:35 STEP 2115 H=2115:00 +CPU= 32.507
  04:34:56 STEP 2116 H=2116:00 +CPU= 21.735
  04:35:17 STEP 2117 H=2117:00 +CPU= 20.779
  04:35:51 STEP 2118 H=2118:00 +CPU= 33.631
  04:36:13 STEP 2119 H=2119:00 +CPU= 21.501
  04:36:35 STEP 2120 H=2120:00 +CPU= 21.969
  04:37:09 STEP 2121 H=2121:00 +CPU= 33.564
  04:37:31 STEP 2122 H=2122:00 +CPU= 21.747
  04:37:53 STEP 2123 H=2123:00 +CPU= 21.334
  04:38:31 STEP 2124 H=2124:00 +CPU= 38.208
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002124
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002124
  04:38:54 STEP 2125 H=2125:00 +CPU= 22.205
  04:39:17 STEP 2126 H=2126:00 +CPU= 22.843
  04:39:51 STEP 2127 H=2127:00 +CPU= 33.749
  04:40:13 STEP 2128 H=2128:00 +CPU= 22.053
  04:40:34 STEP 2129 H=2129:00 +CPU= 20.827
  04:41:08 STEP 2130 H=2130:00 +CPU= 33.869
  04:41:30 STEP 2131 H=2131:00 +CPU= 21.129
  04:41:52 STEP 2132 H=2132:00 +CPU= 21.814
  04:42:25 STEP 2133 H=2133:00 +CPU= 32.957
  04:42:47 STEP 2134 H=2134:00 +CPU= 22.132
  04:43:09 STEP 2135 H=2135:00 +CPU= 22.007
  04:43:49 STEP 2136 H=2136:00 +CPU= 38.987
Moving to projects directory: /var/lib/boinc/slots/11/ICMGGhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/11/ICMSHhq0f+002136
Moving to projects directory: /var/lib/boinc/slots/11/ICMUAhq0f+002136
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002112
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMGGhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMSHhq0f+002124
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12167250/ICMUAhq0f+002124
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0606_1981050100_123_950_12167250_0_r633370543_88.zip
Uploading the intermediate file: upload_file_88.zip
  04:44:10 STEP 2137 H=2137:00 +CPU= 21.284
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:49:13 1494992 22251466 oifs_43r3_ps_0606_1981050100_123_950_12167250_0 7,171,200 54,661 0.0076
09 May 2023 15:48:40 1494992 22251466 oifs_43r3_ps_0606_1981050100_123_950_12167250_0 5,875,200 44,562 0.0076
09 May 2023 15:46:55 1494992 22251466 oifs_43r3_ps_0606_1981050100_123_950_12167250_0 4,579,200 34,396 0.0075
09 May 2023 15:46:45 1494992 22251466 oifs_43r3_ps_0606_1981050100_123_950_12167250_0 3,715,200 28,001 0.0075
09 May 2023 15:45:21 1494992 22251466 oifs_43r3_ps_0606_1981050100_123_950_12167250_0 2,419,200 18,023 0.0074
09 May 2023 15:45:11 1494992 22251466 oifs_43r3_ps_0606_1981050100_123_950_12167250_0 950,400 7,496 0.0079


©2024 cpdn.org