- Contents
- Prerequisites
- New software installation
- Upgrading an existing installation
- Run annalist as a background process
- Accessing Annalist over HTTPS
- Running as a Docker container
- Setting up an Annalist site
- Accessing Annalist
- Create a collection
annalist-manager
commands
- A Unix-like operating system: Annalist has been tested with MacOS 10.11, MacOS 12 and Linux 18.04. Other versions should be usable. (The software did once run on Windows, but the procedure to get it running is somewhat more complicated, and is not fully tested or documented.)
- NOTE: there are problems with SQLite3 on Ubunbtu versions 14.04 and 16.04 (and probably others). Recommended is to use Ubuntu 18.04 or later.
- Python 3
NOTE: As of version 0.5.11, Annalist should be accessed using HTTPS. See discussion below about use of HTTP and HTTPS. TL;DR: to test using HTTP, set environment variable OAUTHLIB_INSECURE_TRANSPORT=1
The following assumes that software is installed under a directory called $WORKSPACE; i.e. Annalist software is installed to $WORKSPACE/annalist. This could be a user home directory.
-
Check the version of python installed on your system. An easy way is to just enter the
python
to see a display like this:$ python Python 3.9.9 (v3.9.9:ccb0e6a345, Nov 15 2021, 13:06:05) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>
(On Linux/Unix systems, typing
python<tab>
may help to show what versions are installed.) -
Go to the workspace directory, create a Python virtual environment and activate it (i.e. make it the current Python environment). This avoids having the Annalist installation stomp over any other Python installation, and makes it very easy to discard if or when it is not required.
cd $WORKSPACE python3 -m venv anenv3 source anenv3/bin/activate python -m pip install --upgrade pip python -m pip install --upgrade certifi python -m pip install --upgrade setuptools pip install annalist
Or, with Python 3.9:
cd $WORKSPACE
/usr/local/bin/python3.9 -m venv --upgrade-deps anenv3
source anenv3/bin/activate
pip install annalist
-
Install the software from PyPI:
pip install annalist
-
Alternatively, obtain a copy of the Annalist distribution kit, e.g. from annalist.net, and copy to a convenient location (e.g., $WORKSPACE/Annalist-0.5.18.tar.gz). Then install it thus:
pip install $WORKSPACE/Annalist-0.5.18.tar.gz
-
Finally, test the installed software:
annalist-manager runtests
The output from this command should look something like this:
$ annalist-manager runtest INFO:annalist_site.settings.runtests:Annalist version 0.5.18 (test configuration) INFO:annalist_site.settings.runtests:SETTINGS_MODULE: annalist_site.settings.runtests INFO:annalist_site.settings.runtests:BASE_DATA_DIR: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/sampledata/data INFO:annalist_site.settings.runtests:CONFIG_BASE: /Users/graham/.annalist/ INFO:annalist_site.settings.runtests:DJANGO_ROOT: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/django INFO:annalist_site.settings.runtests:SITE_CONFIG_DIR: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/annalist_site INFO:annalist_site.settings.runtests:SITE_SRC_ROOT: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root INFO:annalist_site.settings.runtests:TEST_BASE_URI: http://test.example.com/testsite INFO:annalist_site.settings.runtests:DEFAULT_DB_PATH: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/sampledata/data/annalist_test/db.sqlite3 INFO:annalist_site.settings.runtests:DATABASE_PATH: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/sampledata/data/annalist_test/db.sqlite3 INFO:annalist_site.settings.runtests:STATICFILES_DIRS: (('', '/Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/annalist/data/static'), ('images', '/Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/annalist/data/identity_providers/images')) INFO:annalist_site.settings.runtests:ANNALIST_LOG_PATH: /Users/graham/workspace/github/gklyne/annalist/anenv3/lib/python3.9/site-packages/Annalist-0.5.18-py3.9.egg/annalist_root/annalist.log Found 820 test(s). Creating test database for alias 'default'... System check identified no issues (0 silenced). .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 820 tests in 193.291s OK Destroying test database for alias 'default'...
For first-time installations, the Annalist site data will need to be initialized: see section "Setting up an Annalist site" below.
Stop any existing annalist server. Look for background processes running annalist-manager
or django-admin
. There may be up to three separate background processes that need to be stopped.
NOTE: irreversible changes are applied to site and collection data when upgrading, particularly from versions below 0.5.0. It is recommended to make a duplicate copy of the ananlist site data before upgrading the software, This will make it possible to revert to an earlier verson in the event of problems with the newer software. For example, to back up a default "personal" configuration on Linux:
cp -ax ~/annalist_site ~/annalist_site.backup
The installation instructions above ("New software installation") can then be used to update an Annalist software installation, except that step 2 may be skipped if re-using an existing virtual environment.
The annalist user database may need to be initialized or migrated:
annalist-manager initialize [ CONFIG ]
If upgrading from a version 0.1.4 or earlier, recreate an admin user:
annalist-manager defaultadminuser [ CONFIG ]
To update site-wide data for an existing Annalist site, use:
annalist-manager collectstatic [ CONFIG ]
annalist-manager updatesitedata [ CONFIG ]
If any of the Annalist installable collection data is used then these should be re-installed (assuming they have not been modified locally since they were installed); e.g.
annalist-manager installcollection Resource_defs -f [CONFIG]
annalist-manager installcollection Concept_defs -f [CONFIG]
annalist-manager installcollection Journal_defs -f [CONFIG]
(The -f
or --force
option allows the existing collection to be replaced by new defnitions provided by the new Annalist release.)
Existing user collection data will be migrated as it is accessed, but it may be cleaner to migrate all data en masse before startign to access it, thus:
annalist-manager migrateallcollections [ CONFIG ]
Then start the server as before.
e.g.
annalist-manager initialize --personal
annalist-manager defaultadminuser --personal
Creating user admin
Password:
Re-enter password:
annalist-manager collectstatic --personal
annalist-manager updatesitedata --personal
annalist-manager installcollection Resource_defs -f --personal
annalist-manager migrateallcollections --personal
annalist-manager runserver --personal
NOTE: Prior to Annalist version 0.5.14, the annalist-manager runserver
command ran the Django development HTTP server, and operated synchronously; i.e., the annalist-manager runserver
command did not complete until the server itself terminated, commonly by typing CTRL+C. As of version 0.5.14, this is performed by the annalist-manager rundevserver
command. The annalist-manager runserver
command now starts the Annlist application under the gunicorn
HTTP server, and completes as soon as the server is up and running, writing the server process id to stdout (and also to a file in the Annalist site base directory). To stop a running server started in this way, use the command annalist-manager stopserver
.
To run annalist as a long-running background process, use the following command:
nohup annalist-manager runserver
or nohup annalist-manager rundevserver &
(with annalist-manager
configuration options if required.)
To view the server log of a running Annalist instance:
less $(annalist-manager serverlog)
To check for Annalist background processes, and to terminate an Annalist server running as a background process (run from the same user account that started annalist):
For a server started using annalist-manager runserver
:
less $(annalist-manager sitedirectory)/annalist.pid
ps | grep gunicorn
annalist-manager stopserver
For a server started using annalist-manager rundevserver
:
ps x
killall python
As of version 0.5.11, Annalist should be accessed over HTTPS rather than HTTP. The in-built HTTP server does not support HTTPS, so this is achieved by using a standard web server (e.g. Apache httpd or Nginx) to accept incoming HTTPS requests and pass them on to Annalist using local HTTP (using a configuration called "reverse proxying").
This particularly affects the OpenID Connect login, which now fails if an HTTPS connection is not being used (as using HTTP could result in exposure of credentials). To test the software using the in-built development server (i.e. using HTTP rather than HTTPS), use the following command (in a BASH shell):
OAUTHLIB_INSECURE_TRANSPORT=1 python manage.py runserver 0.0.0.0:8000
or
OAUTHLIB_INSECURE_TRANSPORT=1 annalist-manager rundevserver
or
OAUTHLIB_INSECURE_TRANSPORT=1 annalist-manager runserver
Or otherwise set the environment variable OAUTHLIB_INSECURE_TRANSPORT when running the server.
Alternatively, see src/annalist_root/stunnel_dev_https.conf
for using stunnel
to proxy HTTPS requests to HTTP for Django's internal development server.
If using Google authentication, remember to update the login redirect URLs on the developers.google.com
dashboard. Other authentication providers will require similar treatment. The Annalist login redirect URLs end with /annalist/login_done/
.
The Annalist identity provider configuration (typically in ~/.annalist/providers/
must also reflect the correct login redirect URL).
NOTE: Sample configuration files should be created in directory ~/.annalist/config/
by the site initialization procedure (annalist-manager createsite
or annalist-manager updatesite
commands)
NOTE: With Apache 2.4 the authorization directives are changed. The sample configurations have been updated to use Pache 2.4 directioves, and may need to be edited top work with Apache 2.2. Changes apply in the <location>
section:
<location>
# allow from all # Apache 2.2
Require all granted # Apache 2.4
</location>
Prerequisites:
- Apache2 web server installed and running (in default configuration with test page)
Steps to set up HTTPS forwarding. In the following desription, domain name annalist.example.net
is used as an example, and should be replaced with the actual domain name of the server running Annalist.
-
Install LetsEncrypt certificate agent:
apt-get update apt-get install python-certbot-apache
-
The following Apache modules should be installed and enabled (use
a2query -m
to list); usea2enmod
to enable modules (e.g.a2enmod headers, ssl
):headers proxy filter ssl proxy_http autoindex
-
Check firewall configuration. Thje following ports should be enables: 80 (http), 443 (https), 22 (ssh) for access from anywhere. Port 8000 should be enabled for access from the local host address (not just 127.0.0.1). Some comnbination of the following commands might be helpful:
ufw allow ssh ufw allow http ufw allow https ufw allow from <local-IP> to <local-IP> port 8000 ufw enable
If you can't access Annalist with the firewall enabled, look at
/var/log/ufw.log
to see what is being bocked. -
Disable any proxy configuration. This would usually be done by using
a2dissite
to disable any sites for which proxying has been set up. -
Create a proxy configuration for Annalist, if not already done. This can take the form of a file
/etc/apache2/sites-available/annalist.conf
, with content like this:# See: 000-default.host, ports.conf # # See also: https://wiki.apache.org/httpd/CommonMisconfigurations # # Change so that 'annalist.example.net' is your local host name. # e.g. in vim: "%s/annalist.example.net/annalist.yourdomain.org/g" <VirtualHost *:80> ServerName "annalist.example.net" ServerAdmin [email protected] ErrorLog /var/log/apache2/annalist-error.log CustomLog /var/log/apache2/annalist-access.log combined <location /> allow from all </location> RequestHeader set X-Forwarded-Protocol 'http' ProxyPass /annalist http://localhost:8000/annalist ProxyPassReverse /annalist http://localhost:8000/annalist ProxyPass /static http://localhost:8000/static ProxyPassReverse /static http://localhost:8000/static ProxyPreserveHost On </VirtualHost>
-
Generate and install a certificate:
certbot --apache -d annalist.example.net
If all goes well, this will create an additional proxy configuration file
/etc/apache2/sites-available/annalist-le-ssl.conf
containing something like:<IfModule mod_ssl.c> <VirtualHost *:443> ServerName "annalist.example.net" ServerAdmin [email protected] ErrorLog /var/log/apache2/annalist-error.log CustomLog /var/log/apache2/annalist-access.log combined <location /> allow from all </location> RequestHeader set X-Forwarded-Proto 'https' ProxyPass / http://annalist.example.net:8000/ ProxyPassReverse / http://annalist.example.net:8000/ ProxyPreserveHost On SSLCertificateFile /etc/letsencrypt/live/annalist.example.net/cert.pem SSLCertificateKeyFile /etc/letsencrypt/live/annalist.example.net/privkey.pem Include /etc/letsencrypt/options-ssl-apache.conf SSLCertificateChainFile /etc/letsencrypt/live/annalist.example.net/chain.pem </VirtualHost> </IfModule>
(The
RequestHeader
line may need to be edited: it is needed so that the Annalist login logic knows it is being accessed via HTTPS.)If things don't go well, some combination of the following might help:
certbot --help certbot certonly --webroot -d annalist.example.net certbot --apache -d annalist.example.net certbot renew --force-renew
If this doesn't work, look for and remove
SSLCertificateFile
in the base Apache configuration (e.g.,/etc/apache2/sites-available/default-ssl.conf
), and ensure thatcertbot
has added appropriate statements to/etc/apache2/sites-available/annalist-le-ssl.conf
(or whatever name is being used). -
Enable Annalist request forwarding; e.g.
a2ensite annalist # This may turn out to be spurious with HTTPS enforced a2ensite annalist-le-ssl
-
Check the revised configuration and (if OK) restart Apache:
apachectl -t apachectl restart
For a cursory test, try pointing your browser at https://annalist.example.net/: the Annalist site front page, listing available collections, should appear. For a more demanding test, try logging in to Annalist using the Login
button, and selecting Google for the authentication provider.
@@NOTE: these instructions need testing@@
NOTE: Sample configuration files should be created in directory ~/.annalist/config/
by the site initialization procedure (annalist-manager createsite
or annalist-manager updatesite
commands)
Prerequisites:
- nginx web server installed and running (in default configuration with test page)
Steps to set up HTTPS forwarding. In the following description, domain name annalist.example.net
is used as an example, and should be replaced with the actual domain name of the server running Annalist.
-
Install LetsEncrypt certificate agent:
apt-get update
apt-get install python3-certbot-nginx
-
Check firewall configuration. The following ports should be enabled: 80 (http), 443 (https), 22 (ssh) for access from anywhere. Port 8000 should be enabled for access from the local host address (not just 127.0.0.1). Some combination of the following commands might be helpful:
ufw allow ssh ufw allow http ufw allow https ufw allow from <local-IP> to <local-IP> port 8000 ufw enable
If you can't access Annalist with the firewall enabled, look at
/var/log/ufw.log
to see what is being bocked. -
Disable any existing proxy configuration. (This is to prevent proxying from interfering with certbot's verification procedure.)
-
Create a proxy configuration for Annalist, if not already done. This can take the form of a file
/etc/nginx/sites-available/annalist.conf
, with content like this (see also~/.annalist/config/annalist.conf.nginx_example
):server { server_name annalist.example.org; # Non-proxy content location / { root /var/www/html; index index.html index.htm; } # Proxy content matching location ~ /(annalist|static|admin) { proxy_pass http://localhost:8000; proxy_http_version 1.1; proxy_buffering off; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header Host $http_host; proxy_set_header X-Forwarded-protocol $scheme; proxy_set_header X-Forwarded-proto $scheme; } }
-
Generate and install a certificate:
certbot --nginx -d annalist.example.net
When asked, select the option that adds HTTP redirection to HTTPS.
If all goes well, this will update the
/etc/nginx/sites-available/annalist.conf
with details of the Letsencrypt certificate to be used. -
Check the revised configuration and (if OK) restart nginx:
nginx -t service nginx restart
For a cursory test, try pointing your browser at https://annalist.example.net/annalist/
: the Annalist site front page, listing available collections, should appear. For a more demanding test, try logging in to Annalist using the Login
button, and selecting Google for the authentication provider.
See Configuring Annalist to use OpenID Connect.
NOTE the Docker containers created for Annalist are intended for evaluation or local use. They do not have OpenID connect logins configured, and the Annalist server accepts HTTP coinnections on port 8000, including for the login username/password form. You should build your own Annalist docker containers (with HTTPS proxying, OpenID connect and firewall access restrictions configured) for production use. The docker/annalist_dev
files may be used as a starting point for this.
Prerequisite for this option: a Linux operating system with Docker installed. (Also appears to work on Mac with Docker.)
If Annalist Docker containers have been used previously on the host system, the following commamnds ensure you have the latest images:
docker pull gklyne/annalist_site
docker pull gklyne/annalist
If this is the first time Annalist has been run on this system, create a new Annalist site data container:
docker run --name=annalist_site --detach gklyne/annalist_site
Then
docker run --interactive --tty --rm \
--publish=8000:8000 --volumes-from=annalist_site \
gklyne/annalist bash
The remaining commands are executed in the shell environment presented by the docker run
command above.
annalist-manager version
Check the version displayed: I've found Docker sometimes caches older versions and fails up update to the latest available. If necessary, use docker rmi gklyne/annalist
to remove old images from the local cache. If all is well, continue as follows:
If this is the first time Annalist has been run on this system, create a new Annalist site data and database, and define an admin user login:
annalist-manager collectstatic
annalist-manager createsitedata
annalist-manager initialize
annalist-manager defaultadminuser
or, to keep previous annalist collection data:
annalist-manager collectstatic
annalist-manager updatesitedata
NOTE: for configurations which store the database file in the site data area (including the default personal configuration used here), annalist-manager createsitedata
must be run before annalist-manager initialize
, as it requires absence of any previous site data or database files.
When updating an Annalist site, it should not be necessary to run annalist-manager initialize
.
Then, to start the Annalist web server:
annalist-manager runserver
At this point, a browser can be directed to port 8000 (e.g. http://localhost:8000) of the Docker container host to interact with the Annlist system and data.
Existing Analist collection data can be loaded into a new installation, before starting the server. For example, an experimental Digital Music Object, which demonstrates several features of Annalist, can be loaded thus:
cd /annalist_site/annalist_site/c/
git clone https://github.com/gklyne/Carolan_Guitar.git
To run Annalist server as a headless container (no shell):
docker run --detach \
--publish=8000:8000 --volumes-from=annalist_site \
gklyne/annalist \
annalist-manager runserver
From version 0.1.11 onwards, the server logs are located on the annalist_site data volume, and may be viewed by running a second Docker container, thus:
docker run --interactive --tty --rm \
--volumes-from=annalist_site \
gklyne/annalist bash
then
less $(annalist-manager serverlog)
Before setting up an Annalist configuration, there are two issues to be aware of. Or if you just want a quick installation for evaluation purposes, skip ahead to "Initial site setup".
Annalist deployment details are controlled by files in the src/annalist_root/annalist_site/settings
directory. Annalist comes with three pre-defined configurations: development, personal, and shared. The main differences between these are the location of the Annalist site data files, and the location of certain private configuration files. (Other configuration options are possible by defining a new settings file.)
Development: Annalist site data is kept in a directory within the Annalist software source tree, and configuration files are in subdirectory .annalist
of the installing user's home directory.
Personal: Annalist site data is in a subdirectory annalist_site
, and configuration files are in subdirectory .annalist
, of the installing user's home directory.
Shared: Annalist site data is kept in directory /var/annalist_site
, and configuration files are in subdirectory /etc/annalist
. Such an installation will typically require root privileges on the host computer system to complete.
For most purposes, the default (Personal) configuration works just fine. For shared instances, I use this with a dedicated user (e.g. annalist
).
Annalist has been implemented to use federated authentication based on Open ID Connect (http://openid.net/connect/) rather than relying on local user credential management. Using third party authentication services should facilitate integration with single-sign-on (SSO) services, and avoids the security risks assciated with local password storage. Unfortunately, installing and configuring a system to use an OpenID Connect authentication service does take some addtional effort to register the installed application with the authentication service.
Annalist currently supports two user authentication mechanisms: OpenID Connect using Google, and local user login credentials. (Other OpenID Connect providers may also work, but have not been tested.)
Annalist OpenID Connect authentication has been tested with the Google identity service. Instructions for configuring a new installation to work with Google are in Configuring Annalist to use OpenID Connect.
The configuration details for using an OpenID Connect provider are stored in a private area, away from the Annalist source files and site data, since they contain private keying data. A subdirectory providers
of the Annalist configuration directory contains a description file for each supported OpenID Connect provider. New providers may be supported by adding descrtiption files to this directory. The provider description for Google may be a useful example for creating descriptions for other providers. (But be aware that different providers will have different registration procedures, and may require subtlely different forms of configuration information.)
Annalist can also allow users to log in using locally stored credentials, which may be useful for quick evaluation deployments but is not the recommended mechanism for normal operational use.
When installing Annalist, an administration account may be created using the annalist-manager
tool. When logged in to Annalist using this account, the Admin link in the footer of most Annalist pages will allow new user accounts to be created via the Django admin interface. More documentation about using this admin interface is in the The Django Admin Site, which is Chapter 6 of The Django Book.
These instructions use the example of a local user database: these options are not suitable for a full deployment, but are probably the least intrusive to use for early evaluation purposes. Alternatively, for a quickstart, use the Docker container described above.
NOTE: using the development configuration, data files are stored within the software source code tree, and will be removed when the software is updated. Use a personal configuration if you want to preserve any data files you create.
-
The commands must be issued with the annalist python environment activated. If needed, use a command like this (where "anenv2" is replaced with the name used previously to create the python virtual environment):
source anenv2/bin/activate
-
Initialize sitedata:
mkdir ~/annalist_site annalist-manager collectstatic annalist-manager createsitedata
(Don't do this if updating annalist software to use existing site data: use
annalist-manager updatesitedata
instead) -
Initialize user management database
annalist-manager initialize
-
Create admin user
annalist-manager createadminuser
Respond to the prompts with a username, email address and password. The username may be up to 30 characters, and may consist of letters, digits and underscores.
Alternatively, to create a default admin user with name
admin
and email addressadmin@localhost
, use this command:annalist-manager defaultadminuser
As before, enter and re-enter a password when prompted.
-
Start the Annalist server
annalist-manager runserver
or
annalist-manager rundevserver
You should now be able to use a browser to view the Annalist server, e.g. at http://localhost:8000.
NOTE: for configurations which store the database file in the site data area (including the default personal configuration), annalist-manager createsitedata
must be run before annalist-manager initialize
, as it requires absence of any previous site data or database files. When updating an existing Annalist site, it should not be necessary to run annalist-manager initialize
.
The following instructions assume a browser running on the same host as the Annalist service. If a different host is used, replace localhost
with the name or IP address of the host that is running the Annalist server.
-
Browse to annalist server at http://localhost:8000 (replacing
localhost
as needed)An empty list of collections should be displayed, along with some help text:
-
Select the Login item from the top menu bar:
-
Enter the local adminstrator id (e.g.
admin
) in the User Id field, then click on the Local username button: -
Enter the admin user credentials specified previously when creating the Annalist admin user, and click the 'Login' button:
-
Click the Home link on the top menu bar to return to the front page:
Note that the front page now shows text entry boxes and a button for creating a new collection.
At this point, the Admin link in the page footer can be used to create additional local users via the local administrative interface (which is implemented in the underlying Django web application framework). Or just continue straight to create an initial data collection.
This section assumes you are logged in to an Annalist system.
To create a new collection, which is an Annalist unit of administrable and copyable data, enter a short name for the collection (consisting of just letters, digits and/or underscorte ('_'
) characters) and a one-line label or description (which can contain arbitrary characters) into the text boxes presented:
Now click on the 'New' button:
Click on the link in the Id column to view the new collection:
From this screen, you can start to add data to this collection. For more information, see Using Annalist
Use annalist-manager help
for a command summary.