Setting up a Self Hosted Instance

IMPORTANT: This page is a work in progress.

To a certain degree the installation instructions in INSTALLATION.md also apply here. However, when running a server in production a lot of other concerns and small details come up. This guide attempts to clarify and give more instruction for setting up a full production instance of QPixel as a self-hosted service.

Important information about configuration

Way of installation

In ruby, the difference between a config file and a code file is quite blurry. Most "config" files in the config directory are just ruby files with some relatively simple code in them setting a bunch of fields. This makes ruby flexible (you can put code in your configs to make them adapt to the context!), but also a bit more challenging to understand if you are used to just yml files with keys in it. This added complexity also means that the config files are bound to change a bit more than you are used to. For bigger upgrades to QPixel you should expect to have to adapt your config files to match. If we add major dependencies, there is likely to be a so-called initialization config file which needs to be added.

Since you will be configuring your server differently than it comes "out of the box" from QPixels github repo, you need to make a decision on how you want to achieve this. The key problem is that you generally want to stay up to date with QPixel updates and changes, but that you do want potentially vastly different versions of some of the "config" files (which are actually code files). When the QPixel config files get updated, you may want to (or have to) also update your "config" files with those changes. Below we give a few different options for achieving this.

Option 1: Clone and forget (Recommended)

Clone it once, configure everything on the server directly and forget about it. You could even keep git in there to pull occasionally to pull in new changes (only recommended if you are familiar with git).

The main advantage is that it is the most straightforward to set up and configure. If you use this option, you can:

git clone https://github.com/codidact/qpixel
cd qpixel

Option 2: Create a deployment script

It is possible to create a deployment script to deal with turning a folder containing a clone of the repository into a production ready version for your server. This would involve replacing certain files with files from other locations (e.g. replace some configs with symlinks to their properly configured versions).

NOTE: There is a library called Capistrano which attempts to make this process simpler. With a Capistrano definition, it takes care of writing a script and linking the right files while allowing new files to remain unchanged. On a continuous integration server you could have it automatically deploy to your server. A capistrano definition for QPixel could look like this: base deploy configuration, server specific additional configuration. You need quite a bit of capistrano knowledge if you want to go this route, as it requires you to set up the files in the correct way. More information can be found on https://capistranorb.com/.

Option 3: Fork the repository

As an extension of option 1, you could of course also fork the repository and store your changes there. Do take care that you keep your configuration secret or that you sufficiently protect your system where you host your git repository. Perhaps you leave some changes out and combine this option with a deployment script for the last bits. The main advantage is that you can more easily deploy your setup with configs to multiple servers.

Environments

Important to know is that rails (the webserver framework used) uses so-called environments to determine how to run. By default, it will run in a development environment. The development environment has it's own separate set of libraries and settings (which means it will also run to a separate database by default). If you want to do something on production, make sure that before every session you run the following:

export RAILS_ENV=production

You can also prepend every command with RAILS_ENV=production or set this globally for the user that serves the site in its .bashrc (or .zshrc/.fishrc). In this manual we have prepended it everywhere to ensure that all commands copied from here act on the correct environment.

Finalizing

After you have made a choice for how you want to configure QPixel, we can move on to installing everything.

Installation

You have two options for installing QPixel. The first option is to use the existing docker image and adapt it a bit to your needs. This means you don't have to deal with most of the setup hassle. However, the docker image in this repository was intended to set up a quick development instance, so some modifications may need to be necessary to get a performant production setup. See the installation guide for more information on using docker.

The other option is to install ruby and required libraries directly to your server. This also means you will need to run Redis and MySQL on your server (or connect to instances of these running elsewhere).

Installing system libraries

Follow the instructions in the installation guide until Install QPixel to install Ruby and the necessary system libraries.

Installing dependencies - Bundler

As most open source systems, QPixel has a lot of dependencies. QPixel users bundler, which is a package manager for ruby. Install it with:

gem install bundler

There are a few things you should know about bundler that will be important to you.

Installed to the user

By default, Bundler installs its packages to a location specific to the user. This means that you should make sure that the user which will be executing your system, is the one under which you install the packages. Otherwise, you may want to set a custom location to install your dependencies to, and point the executing user to this location.

Packages added to the path

Some dependencies come with executable commands that you can run from the command line. For example, the rails framework adds some essential commands to manage your webserver. By default these packages are added to your path such that you can execute the commands. If this is not the case (for whatever reason), you can prepend any command with bundler exec to execute it in the correct context.

Commands use the latest installed version

When you run a command from the command line without prepending bundler exec, bundler will use the LATEST VERSION of the package installed to execute the command. Even when QPixel defines an older version of the package to be used, it will pick the latest version installed. This can be relevant when you have multiple ruby systems installed on one server, or when you are attempting to rollback after a failed update. To prevent inter-version problems, prepend all commands with bundler exec to ensure that the correct version of the library is used. All commands in this tutorial have bundler exec prepended for this reason.

Finalizing

Having read all this, go ahead and install the production dependencies with:

bundle config set --local without 'development test' 
bundle install

Next we can move on to the configuration.

Configuration

Encryption

Rails uses an encrypted file for storing the most sensitive configuration. For example, if you are using AWS your credentials would go into this encrypted file. By default, the repository contains the encrypted configuration file for Codidact's servers. However, since you don't have the encryption key, this file is useless to you.

One of the important entries in this file is the server secret. This is used as base for anything that needs encrypting, so you should set it to a unique value for your server.

Delete the config/credentials.yml.enc you initially downloaded with the repository.
Run EDITOR=nano RAILS_ENV=production bundle exec rails credentials:edit (you can change EDITOR to vim if you prefer vim)
The secret key base will be set to a newly generated random hex key. If you want to set your own, you can run rails secret to have rails generate a new secret key that you can copy into the file.
Save the file

Database

In config/database.yml rails expects a configuration of your database location and credentials, as well as the same information for redis. By default, the repository comes with a config/database.sample.yml file.

Copy database.sample.yml to database.yml and open it in your editor.
Decide on a database user for QPixel
Create the MySQL user and grant it permission for the database you want to use. For example, in a mysql shell:

CREATE USER qpixel@localhost IDENTIFIED BY 'choose_a_password_here';
GRANT ALL ON qpixel_dev.* TO qpixel@localhost;
GRANT ALL ON qpixel_test.* TO qpixel@localhost;
GRANT ALL ON qpixel.* TO qpixel@localhost;

Put the information in the config file and save.
Specify the location of Redis (or leave the default).
Run RAILS_ENV=production bundle exec rails db:create to create the database. If you get any errors, things may not be configured correctly.
Run RAILS_ENV=production bundle exec rails db:schema:load to create the tables from the db/schema.rb definition (Note for MariaDB, see below)
Run RAILS_ENV=production bundle exec rails r db/scripts/create_tags_path_view.rb
Run RAILS_ENV=production bundle exec rails db:migrate to ensure that your schema contains all required migrations.
QPixel needs timezone definitions to be loaded into MySQL for everything in QPixel to function properly. Follow the instructions at https://github.com/ankane/groupdate#for-mysql .

If you run into issues with multi-byte characters (such as emoji), make sure that your database is set to use utf8mb4.

MariaDB

If you want to use MariaDB instead of MySQL, you can do so. However, you will need to replace all occurrences of utf8mb4_0900_ai_ci with utf8mb4_unicode_ci in db/schema.rb. This is because MariaDB does not support utf8 v9. In practice you will probably not notice any major differences from these collation differences, only where the database is used for searching and sorting and special characters are important for the order.

Redis

Redis is used as semi-permanent cache storage by QPixel (it is a required component). It is essential that Redis is fast(er than your database), so it is recommended to run Redis on the same server as QPixel.

We recommend to configure Redis to have a maximum amount of memory it can use (to an appropriate amount for your server), and to use the least frequently used strategy (allkeys-lfu) to evict keys when Redis is full. The configuration (on Ubuntu) can be found under /etc/redis/redis.conf. You should set the following keys:

maxmemory <appropriate maximum>
maxmemory-policy allkeys-lfu

Storage

In QPixel users can upload files. These files have to go somewhere, and the configuration for this is in config/storage.yml. QPixel uses the ActiveStorage library for managing this, which in turn supports local storage and cloud storage. There is a sample file in config/storage.sample.yml.

Copy the storage.sample.yml to storage.yml
Set up your storage location(s). You can set multiple options here, and we will refer to them later by their name. For example, the following will save to <folder where qpixel is>/storage if local is set as the storage location.

local:
  service: Disk
  root: <%= Rails.root.join('storage') %>
  # root: '/absolute/path/here' if you want an absolute path

In config/environments/production.rb, set:

config.active_storage.service = :name_of_your_selected_storage

to the name of the storage you want to use out of the ones available in storage.yml.

If you need more information on how to configure this, see their official tutorial at https://edgeguides.rubyonrails.org/active_storage_overview.html .

Emails (part 1)

The details of how rails runs your server are in config/environments/production.rb. For most settings you will not have to change the defaults already supplied, but there are a few important ones, one of which is for email.

Set config.action_mailer.delivery_method to :smtp, :sendmail or :ses (Amazon SES).
Set config.action_mailer.default_url_options to { host: 'your.site', protocol: 'https' }
Set config.action_mailer.asset_host to 'https://your.site'

SMTP

If you are using smtp, you want to configure the following settings.

config.action_mailer.smtp_settings = {
  address: 'localhost',       # Default localhost
  port: 25,                   # Default 25
  domain: 'HELO domain',      # HELO Domain. Remove if not used
  user_name: 'user_name',     # Remove if no authentication is used
  password: 'password',       # Remove if no authentication is used
  authentication: :plain,     # Remove if no authentication is used. Options are :plain, :login and :cram_md5
  enable_starttls: false,     # Default false (but autodetect is by default enabled)
  enable_starttls_auto: true, # Autodetect starttls. Default true
  openssl_verify_mode: :none, # Openssl verify mode, either :none or :peer
  tls: true,                  # Enables SMTP/TLS (SMTPS). Remove if you don't want to use or are using starttls.
  open_timeout: 0,            # Nr of seconds to wait while attempting to open a call. Remove if you want default
  read_timeout: 0             # Nr of seconds before timeout a read call. Remove if you want default
}

You can find more details on each of the SMTP parameters at https://guides.rubyonrails.org/configuring.html#config-action-mailer-smtp-settings.

Sendmail

If you are using sendmail with non-default settings, you want to configure the following settings.

config.action_mailer.sendmail_settings = {
  location: '/path/to/sendmail', # Default is /usr/sbin/sendmail
  arguments: '-i' # The command line arguments, default is '-i'
}

SES

If you want to use Amazon SES, set the delivery method to :ses. Next, make sure your SES credentials are set in EDITOR=nano RAILS_ENV=production bundle exec rails credentials:edit. Finally, look at config/initializers/amazon_ses.rb and set it to the correct amazonaws server and signature version.

Emails (part 2) and Sign in and Sign Up

QPixel uses a framework called Devise for handling everything related to registration and authentication. There is an example configuration in config/initializers/devise_example.rb.

Rename devise_example.rb to devise.rb (make sure devise_example.rb does not exist in parallel, otherwise it will override your settings)
Run rails secret and copy the generated secret to config.secret_key (the one in the example is obviously insecure since it is publicly available!)
Set config.mailer_sender to the name and email address you want to appear as sender for account related emails

There are many more settings in this file, all pretty decently explained. We have listed the most important ones that you may want to change below.

Keep users signed in
Set config.remember_for to the amount of time you want to keep a user signed in before they need to give their credentials again.

Password length
Set config.password_length to the range of password lengths you want to accept (e.g. 6..100 is between 6 and 100 inclusive).

Email regexp
Set config.email_regexp to a regex defining the email addresses you want to accept.

Locking accounts
It is possible to enable account locking after a certain number of invalid attempts. See the settings under :lockable and under :recoverable in the file.

SAML
If you want to use SAML sign in, follow the instructions at Setting Up SAML Sign In. We recommend that you first make sure QPixel fully runs on your server before going ahead with enabling the SAML configuration.

Schedule / cronjobs

QPixel uses cron jobs to schedule the emails it sends for subscriptions. The specification of this is in config/schedule.rb. You can modify the times in this file to send out the emails at different moments if you wish. To actually create the corresponding cronjobs from this schedule, run:

RAILS_ENV=production bundle exec whenever --update-crontab

You can also use RAILS_ENV=production bundle exec whenever to just see the created crontab, such that you can set it yourself.

Codidact network

A lot of the codebase expects that you are part of the Codidact network of communities. If you are not, you should alter config/initializers/zz_codidact_sites.rb to have the following content:

Rails.cache.persistent 'codidact_sites', clear: true do
  []
end

By default it will showcase Codidact communities in the navigation bar based on a list, but this will set it to not advertise any communities.

(Optional) Puma

In config/puma.rb you can specify the details of the server process. Here you can set the default amount of threads, port and pidfile of the server process. You can also set all of these with environment variables (PORT, PIDFILE, RAILS_MAX_THREADS, RAILS_MIN_THREADS) when calling rails s or by passing flags to this command instead.

(Optional) Content Security Policy

You can configure a content security policy in config/initializers/content_security_policy.rb. See the instructions in the file for more information.

(Optional) Donations

If you want to use Stripe for donations, set your stripe secrets with EDITOR=nano RAILS_ENV=production rails credentials:edit (stripe_live_secret).

(Optional) Version

By default QPixel uses git in the current repository to generate a version from the commit hash of the last commit. If you do not have the git repository in your server directory, then alter config/initializers/zz_cache_setup.rb to have the following content:

Rails.cache.persistent 'current_commit', clear: true do
  ['1.0', '2023-01-18 13:45:00 +0100']
end

You can set the date here to the date you last updated the system. There may be components which rely on the date being formatted in the way as shown in the example.

Finalizing

Congratulations, you got through the main configuration for the server (at least the things that are in the config folder).

Initializing the site

With an empty database, QPixel cannot be used. There needs to be some initial data in order to create a site and have the ability to configure it (admin account).

Open a rails console with bundle exec rails c -e production. This will open an interactive shell (think Python shell) linked to your server process. Here you can directly execute code as if it would be executed by the server.

Creating a community

QPixel was designed with the idea of running multiple different sites (called communities) from a single server process. Everything in QPixel requires the existance of a community, and relates to a specific community. Thus, you need to create one. In the rails console, enter the following (after changing the name and host accordingly).

Community.create(name: 'Community Name', host: 'mydomain.mysite.com')
Rails.cache.clear

It is important that the community host matches with the domain under which you host the site. Otherwise you will get error 422.

Initializing the database

Next we want to seed our database with the necessary data to get started. For this initial run, we also want to generate the help articles and privacy policy. See https://github.com/codidact/qpixel/blob/develop/INSTALLATION.md#optional-help-topics if you want to alter the help articles, privacy policy or other elements. Then run the following command:

UPDATE_POSTS=true RAILS_ENV=production bundle exec rails db:seed

Finalizing

Now that everything is set up, the remainder should be configurable from the site itself. At this moment it should be possible to use bundle exec rails s -e production to start our site without errors, but we can probably not reach it yet. The next step is to put it behind a NGINX or apache proxy to actually make it reachable.

Serving the site

Next, it is up to you to choose how to actually server the site. Most people run rails' webserver behind Apache or NGINX where Apache/NGINX performs a reverse proxy. For this tutorial we will assume you are using NGINX, but a similar configuration should be required for Apache. You can find numerous tutorials online by searching for "rails puma with nginx" or "rails puma with apache" (Puma is the name of the webserver used by default by QPixel).

Service

You need to make sure that rails runs as a service. You probably also want to make sure the service starts with your server.

If you are using systemd (Ubuntu), you can create a file at /etc/systemd/system/qpixel.service:

[Unit]
Description=QPixel
After=network.target
After=mysql.service
Requires=mysql.service

[Service]
User=<user running the service>
Group=<group running the service>
WorkingDirectory=<location for qpixel server files, e.g. /var/www/qpixel>
ExecStart=bundle exec rails s -e production
TimeoutStartSec=3600
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Next enable the service systemctl enable qpixel and start the service systemctl start qpixel.

Assets

By default, codidact usees NGINX to serve the assets from the public folder. It is also possible to have puma serve these files. If you want to do so, we recommend that you set a cache header on them to ensure that your server is not flooded with requests for simple assets. In config/environments/production.rb, you set the following to serve them and have these files be cached:

config.public_file_server.enabled = true
config.public_file_server.headers = {
  'Cache-Control' => "public, max-age=#{5.days.to_i}"
}

You can alter the max age from 5 to a different number of days if you so desire.

NGINX config

You need to set NGINX to a reverse proxy. Additionally, you need to set the HOST header such that rails knows what site the request came from. Create a normal NGINX configuration and add the following for the reverse proxy:

proxy_pass http://localhost:3000;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;

If you bind rails to a different address or port, you need to configure that here.

Finalizing

Assuming that your service is started (systemctl start qpixel) and your nginx configured, you should now be able to visit your site in your browser!

On-Site configuration

First we will need to create an administrator account.

On the site, use Sign Up to create an account.
Open a rails console (bundle exec rails c -e production) and run the following:

User.last.update(confirmed_at: DateTime.now, is_global_admin: true)

Your account is now confirmed and you can sign in.

Configuring your site

Follow the instructions in the installation manual starting from New site setup.

Finalizing

Congratulations, you should now have a fully functioning QPixel instance! 🎉

However, you may notice some things which are not quite as you'd like them. Some parts may imply you are part of the Codidact network, link to Codidact sites erroneously or you may see some emails getting sent from the wrong address. Let's fix that.

"Configuration" (part 2)

Unfortunately, there are still some hardcoded references to the Codidact network or even hardcoded email addresses over the codebase. There is an effort to make these configurable and to remove these references, but this is a slow process. This manual will go over the changes that were necessary on January 19th 2023 to remove these references.

Mailers

The folder app/mailers contains the definitions for emails that are sent from the system. This also contains the (incorrect) from addresses.

In most of these files, there is a line at the top default from: '...' . You should change these to your preferred email address in the following format:

default from: 'Name to display <[email protected]>'

Additionally, there are lines starting with mail ... which can have from: '...', subject: '...' and to: '...'. You should also modify these to your needs.

TODO

Tuning performance

Performance metrics

When signed in as an admin user, you get a flamegraph analysis of the load times of the different components on a page. There will be a small [...] in the top left corner of the page which displays this information to you. You can use this to identify why pages are loading slowly.

Puma

In config/puma.rb you can tune the webserver. If you have a lot of users, you may want to create more threads for handling requests, or use a different concurrency mode. There are comments in the file for more information, and if you need more, see their GitHub repository at https://github.com/puma/puma .

Troubleshooting

Error 422 No community record matching Host `=<hostname>`

Usually this is caused by the host you set when creating a community does not match the domain from which you are accessing the site (no community you created matches the current domain). You can alter the host of your community in a rails console (bundle exec rails c -e production) with the following code:

Community.first.update(host: 'correct.host.for.my.site.com')

If you have multiple communities, you can retrieve them by name to update the correct one.

Community.find_by(name: '...').update(host: 'correct.host')

It is also possible that your NGINX or Apache proxy does not pass the domain of the original request along. Make sure the X-Forwarded-For header is set to the original domain of the request.

JS/CSS is not being loaded

Your public assets are not being served. Either configure NGINX/Apache to serve the contents of the public folder, or enable public serving in Rails (see Assets).

Another reason may be that you need to precompile the rails assets. You can run RAILS_ENV=production bundle exec rails assets:precompile to do so. After updating QPixel, be sure to run this command again.

Finally, make sure to restart rails. If you are using systemd with the configuration above, you can use systemctl restart qpixel.

Updating QPixel

When updating to a new version of QPixel there are a few things to think about. First, you may need to merge changes to config files with your versions. Next, you will need to update the packages to the correct version:

bundle install

If any database changes were made, you also need to run migrations:

Backup your database
Run RAILS_ENV=production bundle exec rails db:migrate

Additionally, you may need to recompile assets with RAILS_ENV=production bundle exec rails assets:precompile.

Finally, restart rails. If you are running it as a service in systemd, you can use systemctl restart qpixel.

Setting up a Self Hosted Instance

Important information about configuration

Way of installation

Option 1: Clone and forget (Recommended)

Option 2: Create a deployment script

Option 3: Fork the repository

Environments

Finalizing

Installation

Installing system libraries

Installing dependencies - Bundler

Installed to the user

Packages added to the path

Commands use the latest installed version

Finalizing

Configuration

Encryption

Database

MariaDB

Redis

Storage

Emails (part 1)

SMTP

Sendmail

SES

Emails (part 2) and Sign in and Sign Up

Schedule / cronjobs

Codidact network

(Optional) Puma

(Optional) Content Security Policy

(Optional) Donations

(Optional) Version

Finalizing

Initializing the site

Creating a community

Initializing the database

Finalizing

Serving the site

Service

Assets

NGINX config

Finalizing

On-Site configuration

Configuring your site

Finalizing

"Configuration" (part 2)

Mailers

TODO

Tuning performance

Performance metrics

Puma

Troubleshooting

Error 422 No community record matching Host =<hostname>

JS/CSS is not being loaded

Updating QPixel

Clone this wiki locally

Error 422 No community record matching Host `=<hostname>`