fix non legacy build/docker issues plus dev to rpm install mechanism #…

…1989

Fix a number of the remaining build and docker issues concerning the
proposed move to openSUSE as an upstream linux ditro base. Includes a
fix to facilitate moving from a developer (source) install to an rpm
based one: currently this only supports our existing legacy CentOS base,
pending the instantiation of an openSUSE rpm build backend along with
related distro aware repository config code changes.

Docker specific notes:
Many more modern dockerd invocations require a number of command line
arguments. Previously we passed, from the docker.service file, only one:
our --data-root target. The included modifications allows for more
custom or distro specific requirements to be met via accommodation of
any number of arguments (unfiltered). All prior Rockstor specific
dockerd arguments are preserved and applied as before.

Summary:
- Add dependency on python ‘distro’ library.
- Store build system distro info in django settings, the assumption here
is we build on our target distro: normally the case.
- Add distro UI element, uses prior 2 items.
- Normalise prior UI subheader linux info formatting.
- Remove prior incorrect data_collector code comment.
- Selectively run postgresql-setup (legacy) or initdb (non legacy) in
initrock.
- Update psycopg2 from 2.6 to 2.7.4.
- Normalise on direct paths for commands: avoids redundant fs
redirection ie in CentOS root we have “/bin -> /usr/bin" and
"/sbin - > /usr/sbin"; as these dir links are not found in our non
legacy base move all hard wired command paths using them to their
canonical reference.
- Use Django settings for a selection of variably located
(distro specific) command paths: again with the assumption that we build
on our target distro.
- Fix version indicator and software update page display for dev
(source) installs.
- Fix dev (source install) to rpm install transition mechanism -
necessarily considered as a re-install so db is wiped during the
transition. Note that this, in part, involved the addition of an
explicit 'yum install rockstor' command during update, along with
ensuring that initrock is re-run on next rockstor.service start.
- Add distro aware docker.service template file selection based on
distro.id(); moving fully to a live edit (during Rock-on service enable)
rather than build time customization: ie to accommodate for our docker
wrapper redirect and it’s consequent requirement for NofityAccess=all
for Type=notify docker configs. Both included openSUSE templates are
taken from their respective distro default installs of docker-ce.
- Establish docker-generic.service failover config for unknown distro
ids taken from default upstream docker-ce 18.09 CentOS example.
- Enhance docker wrapper to pass additional arguments to dockerd.
- Minor additional rock-ons-root config exception logging.
- Catch and log harmless reboot/shutdown command exceptions with rc=-15.
The exception log reports from these Web-UI initiated events are
misleading as they suggest malfunction where there is none as both
commands execute as expected with: out='', err='', and rc=-15.

base-buildout.cfg

@@ -61,7 +61,7 @@ command =
 	postgresql-server postgresql-devel kernel-ml btrfs-progs rsync \
 	nfs-utils avahi netatalk smartmontools net-tools sos hdparm \
 	postfix cyrus-sasl-plain yum-cron nano usbutils pciutils shellinabox \
-	epel-release cryptsetup docker-ce
+	epel-release cryptsetup docker-ce python-distro
 
 [rpm-deps-nut]
 recipe = plone.recipe.command
@@ -99,7 +99,6 @@ gunicorn = 19.7.1
 supervisor = 3.0b1
 python = 2.7.3
 djangorecipe = 1.9
-psycopg2 = 2.6
 
 [django]
 recipe = djangorecipe
@@ -145,9 +144,12 @@ on_update = true
 cmds = ${buildout:directory}/bin/django collectstatic --noinput -i admin -v 0
 
 [docker-conf]
+# Consider inline sed of system's /usr/lib/systemd/system/docker.service
+# that way we pick up new versions on each build.
+# Or depricate and rely on docker_service.py to inline edit and assert.
 recipe = collective.recipe.template
 input = ${buildout:directory}/conf/docker.service.in
-output = ${buildout:directory}/conf/docker.service
+output = ${buildout:directory}/conf/docker-rockstor.service
 
 [rockstor-systemd-conf]
 recipe = collective.recipe.template

conf/django-hack

@@ -24,7 +24,7 @@ sys.path[0:0] = [
     join(base, 'eggs/oauthlib-1.0.1-py2.7.egg'),
     join(base, 'eggs/psutil-3.3.0-py2.7-linux-x86_64.egg'),
     join(base, 'eggs/psycogreen-1.0-py2.7.egg'),
-    join(base, 'eggs/psycopg2-2.6-py2.7-linux-x86_64.egg'),
+    join(base, 'eggs/psycopg2-2.7.4-py2.7-linux-x86_64.egg'),
     join(base, 'eggs/python_engineio-1.0.3-py2.7.egg'),
     join(base, 'eggs/python_socketio-1.6.0-py2.7.egg'),
     join(base, 'eggs/pytz-2014.3-py2.7.egg'),

conf/docker-distroid-notes.service.txt

@@ -0,0 +1,13 @@
+Files with a name pattern of: docker-distroid.service
+
+where distroid = distro.id()
+
+are currently hand copied from their distro docker package origin, usually:
+/usr/lib/systemd/system/docker.service
+
+src/rockstor/smart_manager/views/docker_service.py then stream edits them.
+
+See docker_service.py for pre-instantiation edit details.
+
+Auto edit is performed on Docker service enable event within Web-UI and
+attempts to honour all existing dockerd arguments.

conf/docker-generic.service

@@ -0,0 +1,46 @@
+[Unit]
+Description=Docker Application Container Engine
+Documentation=https://docs.docker.com
+BindsTo=containerd.service
+After=network-online.target firewalld.service
+Wants=network-online.target
+
+[Service]
+Type=notify
+# the default is not to use systemd for cgroups because the delegate issues still
+# exists and systemd currently does not support the cgroup feature set required
+# for containers run by docker
+ExecStart=/usr/bin/dockerd -H unix://
+ExecReload=/bin/kill -s HUP $MAINPID
+TimeoutSec=0
+RestartSec=2
+Restart=always
+
+# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
+# Both the old, and new location are accepted by systemd 229 and up, so using the old location
+# to make them work for either version of systemd.
+StartLimitBurst=3
+
+# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
+# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
+# this option work for either version of systemd.
+StartLimitInterval=60s
+
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+
+# Comment TasksMax if your systemd version does not supports it.
+# Only systemd 226 and above support this option.
+TasksMax=infinity
+
+# set delegate yes so that systemd does not reset the cgroups of docker containers
+Delegate=yes
+
+# kill only the docker process, not all processes in the cgroup
+KillMode=process
+
+[Install]
+WantedBy=multi-user.target

conf/docker-opensuse-leap.service

@@ -0,0 +1,35 @@
+[Unit]
+Description=Docker Application Container Engine
+Documentation=http://docs.docker.com
+After=network.target containerd.socket containerd.service lvm2-monitor.service SuSEfirewall2.service
+Requires=containerd.socket containerd.service
+
+[Service]
+EnvironmentFile=/etc/sysconfig/docker
+
+# While Docker has support for socket activation (-H fd://), this is not
+# enabled by default because enabling socket activation means that on boot your
+# containers won't start until someone tries to administer the Docker daemon.
+Type=notify
+ExecStart=/usr/bin/dockerd --containerd /run/containerd/containerd.sock --add-runtime oci=/usr/sbin/docker-runc $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS
+ExecReload=/bin/kill -s HUP $MAINPID
+
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this property.
+TasksMax=infinity
+
+# Set delegate yes so that systemd does not reset the cgroups of docker containers
+# Only systemd 218 and above support this property.
+Delegate=yes
+
+# This is not necessary because of how we set up containerd.
+#KillMode=process
+
+[Install]
+WantedBy=multi-user.target

conf/docker-opensuse-tumbleweed.service

@@ -0,0 +1,34 @@
+[Unit]
+Description=Docker Application Container Engine
+Documentation=http://docs.docker.com
+After=network.target lvm2-monitor.service SuSEfirewall2.service
+
+[Service]
+EnvironmentFile=/etc/sysconfig/docker
+
+# While Docker has support for socket activation (-H fd://), this is not
+# enabled by default because enabling socket activation means that on boot your
+# containers won't start until someone tries to administer the Docker daemon.
+Type=notify
+ExecStart=/usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS
+ExecReload=/bin/kill -s HUP $MAINPID
+
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this property.
+TasksMax=infinity
+
+# Set delegate yes so that systemd does not reset the cgroups of docker containers
+# Only systemd 218 and above support this property.
+Delegate=yes
+
+# This is not necessary because of how we set up containerd.
+#KillMode=process
+
+[Install]
+WantedBy=multi-user.target

conf/docker.service.in

@@ -1,7 +1,7 @@
 [Unit]
 Description=Docker Application Container Engine
 Documentation=http://docs.docker.com
-After=network.target docker.socket rockstor-bootstrap.service
+After=network.target docker.socket
 Requires=docker.socket
 
 [Service]

conf/settings.conf.in

@@ -19,6 +19,7 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 
 # Django settings for Rockstor project.
 import os
+import subprocess, distro
 
 DEBUG = ${django-settings-conf:debug}
 TEMPLATE_DEBUG = DEBUG
@@ -416,4 +417,23 @@ TASK_SCHEDULER = {
 	       'max_log': 100 #max number of task log entries to keep
 }
 
-OAUTH2_PROVIDER_APPLICATION_MODEL = 'oauth2_provider.Application'
+OAUTH2_PROVIDER_APPLICATION_MODEL = 'oauth2_provider.Application'
+
+# Setup OS specific command paths via 'which cmd' calls
+# N.B. this method will not work with an alias, ie in CentOS
+# which ls
+# alias ls='ls --color=auto'
+#         /usr/bin/ls
+# The following have been tested in CentOS, openSUSE Leap15, and Tumbleweed
+UDEVADM = subprocess.check_output(["which", "udevadm"]).rstrip()
+SHUTDOWN = subprocess.check_output(["which", "shutdown"]).rstrip()
+CHKCONFIG_BIN = subprocess.check_output(["which", "chkconfig"]).rstrip()
+
+# Establish our OS base id, name, and version:
+# Use id for code path decisions. Others are for Web-UI display purposes.
+# Examples given are for CentOS Rockstor variant, Leap 15, and Tumblweed.
+OS_DISTRO_ID = distro.id()  # rockstor, opensuse-leap, opensuse-tumbleweed
+OS_DISTRO_NAME = distro.name()  # Rockstor, openSUSE Leap, openSUSE Tumbleweed
+# Note that the following will capture the build os version.
+# For live updates (running system) we call distro.version() directly in code.
+OS_DISTRO_VERSION = distro.version()  # 3, 15.0 ,20181107

setup.py

@@ -76,12 +76,13 @@
         'mock == 1.0.1',
         'psutil == 3.3.0',
         'psycogreen == 1.0',
-        'psycopg2 == 2.6',
+        'psycopg2 == 2.7.4',
         'python-socketio == 1.6.0',
         'pytz == 2014.3',
         'pyzmq == 15.0.0',
         'requests == 1.1.0',
         'six == 1.10.0',
+        'distro',
     ]
 
 )

src/rockstor/fs/btrfs.py

@@ -34,12 +34,12 @@
 
 logger = logging.getLogger(__name__)
 
-MKFS_BTRFS = '/sbin/mkfs.btrfs'
-BTRFS = '/sbin/btrfs'
-MOUNT = '/bin/mount'
-UMOUNT = '/bin/umount'
+MKFS_BTRFS = '/usr/sbin/mkfs.btrfs'
+BTRFS = '/usr/sbin/btrfs'
+MOUNT = '/usr/bin/mount'
+UMOUNT = '/usr/bin/umount'
 DEFAULT_MNT_DIR = '/mnt2/'
-RMDIR = '/bin/rmdir'
+RMDIR = '/usr/bin/rmdir'
 QID = '2015'
 # The following model/db default setting is also used when quotas are disabled.
 PQGROUP_DEFAULT = settings.MODEL_DEFS['pqgroup']

src/rockstor/scripts/docker_wrapper.py

@@ -23,16 +23,30 @@
 
 DOCKERD = '/usr/bin/dockerd'
 
+ROCKSTOR_DOCKER_OPTS = [
+    '--log-driver=journald',
+    '--storage-driver', 'btrfs',
+    '--storage-opt', 'btrfs.min_space=1G']
+
 
 def main():
-    mnt_pt = sys.argv[1]
+    # We expect the last element of our argument list to be the mount point as
+    # docker_service.py formats it that way.:
+    mnt_pt = sys.argv[-1]
+    # N.B. sys.argv[0] is name of script itself and always present.
+    system_docker_opts = []
+    if len(sys.argv) > 2:
+        # We have at least 1 additional argument passed so extract it/them ie:
+        # [script-name, additional-arg, mount-point]
+        # we extract additional-arg (or it's plural counterpart) as a list.
+        system_docker_opts = sys.argv[1:-1]
     sname = mnt_pt.split('/')[-1]
     try:
         so = Share.objects.get(name=sname)
         mount_share(so, mnt_pt)
     except Exception as e:
         sys.exit('Failed to mount Docker root(%s). Exception: %s' %
                  (mnt_pt, e.__str__()))
-    run_command([DOCKERD, '--log-driver=journald', '--storage-driver',
-                 'btrfs', '--storage-opt', 'btrfs.min_space=1G', '--data-root',
-                 mnt_pt])
+    cmd = [DOCKERD] + ROCKSTOR_DOCKER_OPTS + system_docker_opts + \
+          ['--data-root', mnt_pt]
+    run_command(cmd)

src/rockstor/scripts/initrock.py

@@ -347,7 +347,17 @@ def main():
             logger.debug('Deleting /var/lib/pgsql/data')
             shutil.rmtree('/var/lib/pgsql/data')
         logging.info('initializing Postgresql...')
-        run_command(['/usr/bin/postgresql-setup', 'initdb'])
+        # Conditionally run this only if found (CentOS/RedHat script)
+        if os.path.isfile('/usr/bin/postgresql-setup'):
+            logger.debug('running postgresql-setup initdb')
+            # Legacy (CentOS) db init command
+            run_command(['/usr/bin/postgresql-setup', 'initdb'])
+        else:
+            ## In eg openSUSE run the generic initdb from postgresql##-server
+            if os.path.isfile('/usr/bin/initdb'):
+                logger.debug('running generic initdb on {}'.format(pg_data))
+                run_command(
+                    ['su', '-', 'postgres', '-c', '/usr/bin/initdb', pg_data])
         logging.info('Done.')
         run_command([SYSCTL, 'restart', 'postgresql'])
         run_command([SYSCTL, 'status', 'postgresql'])

src/rockstor/smart_manager/data_collector.py

@@ -51,6 +51,7 @@
 from system.services import service_status  # noqa E402
 from cli.api_wrapper import APIWrapper  # noqa E402
 from system.pkg_mgmt import (update_check, yum_check)  # noqa E402
+import distro
 import logging  # noqa E402
 logger = logging.getLogger(__name__)
 
@@ -838,6 +839,7 @@ class SysinfoNamespace(RockstorIO):
 
     start = False
     supported_kernel = settings.SUPPORTED_KERNEL_VERSION
+    os_distro_name = settings.OS_DISTRO_NAME
 
     # This function is run once on every connection
     def on_connect(self, sid, environ):
@@ -857,6 +859,7 @@ def on_connect(self, sid, environ):
         self.spawn(self.prune_logs, sid)
         self.spawn(self.send_localtime, sid)
         self.spawn(self.send_uptime, sid)
+        self.spawn(self.send_distroinfo, sid)
         self.spawn(self.shutdown_status, sid)
         self.spawn(self.pool_degraded_status, sid)
         self.spawn(self.pool_dev_stats, sid)
@@ -868,11 +871,18 @@ def on_disconnect(self, sid):
         self.start = False
 
     def send_uptime(self):
-        # Seems redundant
+
         while self.start:
             self.emit('uptime', {'key': 'sysinfo:uptime', 'data': uptime()})
             gevent.sleep(60)
 
+    def send_distroinfo(self):
+        while self.start:
+            data = {'distro': self.os_distro_name, 'version': distro.version()}
+            self.emit('distro_info',
+                      {'key': 'sysinfo:distro_info', 'data': data})
+            gevent.sleep(600)
+
     def send_localtime(self):
 
         while self.start: