durhack-deployer

Current problems

• Celery is not suited to async operations
• Celery is not fit-for-purpose. it is really meant for handling a high volume of small messages. We have a small volume of larger messages.

Ideas

• Use the filesystem as our queue instead of Celery; we can use a file-watcher to detect changes (additions to the queue)
  • i.e. a directory roughly corresponds to a 'queue'. the files within the directory are the queue entries
  • each queue entry can be named with an epoch timestamp such that lexicographic ordering should yield the correct order of events
• Each queue worker can run on an asyncio event loop
  • 1 worker+queue that handles all GitHub events and forwards them to the appropriate queue, or voids them if
  • 1 worker+queue per repository deployment (i.e. durhack and durhack-staging should be distinct)
• Use redis-lock where inter-process locking is necessary
• Use PM2 to manage the Flask app & worker python processes

Stack/Tooling

• Most dependencies are explained by comments in Pipfile, which is analogous to package.json in a JavaScript project. The program that interacts with / restores environment using the Pipfile is pipenv
• The main HTTP server is a Flask application listening on port 3400
  • Flask implements the WSGI
  • This means that in production, you shouldn't just use the Flask development server. You should choose a WSGI server (options) and use that to run your WSGI (Flask) application.
  • Joe chose uWSGI because they had documentation on managing their server/your apps using a systemd service
• The application needs to persist some information (specifically, GitHub event IDs of previously processed events)
  • it uses sqlalchemy, a Python ORM, to access a postgres database
  • it uses alembic, a database migration tool, to manage changes to the database (hopefully) without losing access to data
• (Currently unused - not necessary, keeping these instructions just-in-case) To facilitate 'locking' inter-process shared resource to prevent concurrent access, we want an in-memory database. Joe chose dragonfly, which is "a drop-in Redis replacement".
  • copy the download link for a dragonfly executable archive (usually, x86-64.tar.gz)
  • sudo mkdir /opt/dragonfly/ && cd /opt/dragonfly - create /opt/dragonfly and cd into it
  • sudo wget [paste link] - download the archive you copied the link for
  • sudo mkdir ./dragonfly-v1.23.2 - create a directory for the version you downloaded (get version number from GitHub)
  • sudo tar -xf dragonfly-x86_64.tar.gz -C ./dragonfly-v1.23.2 - unpack the archive into the directory you created
  • sudo rm dragonfly-x84_64.tar.gz - remove the archive as we don't need it anymore
  • sudo ln -s ./dragonfly-v1.23.2 ./dragonfly-current - create a symlink dragonfly-current which points to dragonfly-v1.23.2
  • sudo ln -s /opt/dragonfly/dragonfly-current/dragonfly-x86_64 /usr/local/bin/dragonfly - create a symlink dragonfly in /usr/local/bin which points to the binary dragonfly-x84_64 in /opt/dragonfly/dragonfly-current
  • dragonfly --help to test that dragonfly has successfully been installed
  • sudo apt install redis-tools so we can use redis-cli as a database client for dragonfly
  • Create a new user dragonfly with homedir /var/lib/dragonfly
    • sudo adduser dragonfly --group --system --disabled-password --home /var/lib/dragonfly --shell /bin/bash
    • cd /var/lib && sudo chmod o+rx dragonfly - to others (o), add (+) read (r) and execute (x) perms on dragonfly's home directory
  • Create a service file at /etc/systemd/system/dragonfly.service:

    [Unit]
    Description=Dragonfly In-Memory Data Store
    After=network.target
    
    [Service]
    User=dragonfly
    Group=dragonfly
    ExecStart=/usr/local/bin/dragonfly --dir /var/lib/dragonfly
    ExecStop=/usr/bin/redis-cli shutdown
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    

  • sudo systemctl daemon-reload so that systemd will check for new service units,
  • sudo systemctl start dragonfly to start dragonfly as a daemon (background process)
• About the durhack-deployer user

  • Q: Why don't we just use root? A: anything that does not need to run as root, should not run as root. Setting up fine-grained access control protects us from ourselves and malicious actors.

  • That said, permissions are going to be tricky ...

    • the user needs to be able to run the certbot CLI (which requires root privileges)
    • the user needs to be able to run systemctl reload nginx as root (but not any other systemctl commands)
    • the user needs to be able to read from/write to the /etc/nginx/conf.d directory

    We will have to make use of a few systems to satisfy all constraints.

  • Create the user (and group):

    sudo adduser durhack-deployer --disabled-password --home /home/durhack-deployer

    • --disabled-password: do not set a password, but still permit login (for example via sudo -u or SSH)
    • --home home: set the home directory ($HOME, ~) for the user to home

    (you can leave all the user information fields blank, just keep pressing Enter until 'Is the information correct?')

  • We create a file in /etc/sudoers.d which will be included by /etc/sudoers.

    $ cd /etc/sudoers.d
    /etc/sudoers.d$ sudo touch durhack-deployer`

  • Add a directive granting durhack-deployer access to certbot

    /etc/sudoers.d$ sudo bash -c "cat >> durhack-deployer"
    # allow `durhack-deployer` to run `certbot` as `root` without a password and with arbitrary arguments
    durhack-deployer ALL=(ALL) NOPASSWD: /usr/bin/certbot
    ^C

  • Add a directive granting durhack-deployer access to systemctl reload nginx

    /etc/sudoers.d$ sudo bash -c "cat >> durhack-deployer"
    # allow `durhack-deployer` to run `systemctl` as `root` without a password and only with the exact arguments `reload nginx`
    durhack-deployer ALL=(ALL) NOPASSWD: /usr/bin/systemctl reload nginx
    ^C

  • We edit the file access control list (FACL) of /etc/nginx/conf.d to permit durhack-deployer to read from/write to its files.

    $ sudo apt install acl -y
    ...
    $ cd /etc/nginx
    /etc/nginx$ # modify the default ACL for the directory (only affects newly created files)
    /etc/nginx$ sudo setfacl --default -m user:durhack-deployer:rw conf.d
    /etc/nginx$ # modify the ACL of existing files
    /etc/nginx$ sudo setfacl -R -m user:durhack-deployer:rw conf.d
    /etc/nginx$ # modify the ACL of the directory to allow execution (necessary for creation/deletion of files within the directory)
    /etc/nginx$ sudo setfacl -m user:durhack-deployer:rwx conf.d

  • We are done! sudo -u durhack-deployer -i to start a login shell as durhack-deployer, ctrl+D to logout.