This describes the standard file-media relationship in Islandora. There is an alternative method of arranging files and their derivatives -which we call the "Multi-file media" method.
+which we call the "Multi-file media" method.To experience the full Islandora Starter Site, it requires access to external services such as Solr,
Fedora, Alpaca, and Matomo. It is therefore suggested to deploy the Starter Site using one of our
-two deployment platforms: ISLE-DC (using the make starter or make starter_dev commands), or
-the Islandora Playbook (using the starter (default) or starter_dev option in the Vagrantfile).
make starter or make starter_dev commands), ISLE Site Template, or
+the Islandora Playbook (using the starter (default) or starter_dev option in the Vagrantfile).
In order to generate that custom Drupal image we need to set up a development environment. You will do this +
In order to generate that custom Drupal image we need to set up a development environment. You will do this on your development computer, rather than your production server.
Once your development site is set up you will need somewhere to store your custom Drupal image. You should create a private repository in your container registry of choice (Dockerhub, GitHub, GitLab, etc.)
@@ -2886,7 +2886,7 @@If you don't already have a Drupal site, you'll be given a basic setup using Drupal 9 and the +
If you don't already have a Drupal site, you'll be given a basic setup using Drupal 10 and the Islandora Starter Site.
If you do already have a Drupal site, use git to clone it into place as the codebase folder.
cd /path/to/isle-dc
@@ -2666,7 +2666,7 @@ Getting Startedmake starter
to install the Drupal site in your codebase folder and spin up all the other containers with it.
Enjoy your Islandora instance! Check out the basic usage documentation to see +
Enjoy your Islandora instance! Check out the basic usage documentation to see
all the endpoints that are available and how to do things like start and stop Islandora. Your passwords,
including the Drupal admin password, can be found in the secrets/live directory after you run make starter.
Drupal updates are performed through composer on your development site. Once the modules have been added/removed/updated, your composer.json and composer.lock files can be checked into your git repository and you can rebuild your production Drupal container with the new files.
Composer commands need to run in your Drupal container. For example:
+Composer commands need to run in your Drupal container. For example:
docker compose exec drupal-dev composer update -W
Or
docker compose exec drupal-dev composer require 'drupal/islandora:^2.11'
Needs Maintenance
-The manual installation documentation is in need of attention. We are aware that some components no longer work as documented here. If you are interested in helping us improve the documentation, please see Contributing.
+The manual installation documentation is in need of attention. We are aware that some components no longer work as documented here. If you are interested in helping us improve the documentation, please see Contributing.
After all of the above pieces are in place, installed, configured, started, and otherwise prepared, the last thing we need to do is to finally configure the front-end Drupal instance to wire all the installed components together.
settings.php'driver' => 'pgsql',
+
After:
-
'driver' => 'pgsql',
+'driver' => 'mysql',
);
$settings['trusted_host_patterns'] = [
@@ -2820,7 +2820,7 @@ settings.phpIslandora¶
Skip this by using the Islandora Starter Site
-The Islandora Starter Site, which was presented as an option in the "Installing Composer, Drush, and Drupal" step,
+
The Islandora Starter Site, which was presented as an option in the "Installing Composer, Drush, and Drupal" step,
installs Islandora and other modules and configures them, allowing you to skip this section. You may want to use this manual method in the case where you want
to pick and choose which Islandora modules you use.
@@ -2910,7 +2910,7 @@ Enabling EVA ViewsJuly 15, 2024
+ August 16, 2024
diff --git a/pr-preview/pr-2331/installation/manual/installing-alpaca/index.html b/pr-preview/pr-2331/installation/manual/installing-alpaca/index.html
index dbca81ff5..8c1b61329 100644
--- a/pr-preview/pr-2331/installation/manual/installing-alpaca/index.html
+++ b/pr-preview/pr-2331/installation/manual/installing-alpaca/index.html
@@ -496,7 +496,7 @@
-
+
8.x-2.0 Release Notes
@@ -1097,6 +1097,26 @@
Installing ActiveMQ
+
+
@@ -2655,6 +2675,26 @@
Installing ActiveMQ
+
+
@@ -2760,7 +2800,8 @@ In this section, we will install:Islandora/Alpaca, Java middleware that handle communication between various components of Islandora.
Installing ActiveMQ¶
-In our case, the default installation method for ActiveMQ via apt-get will suffice.
+Some users have been able to install ActiveMQ from the standard package repositories. Others, however, have needed to install it manually.
+Option 1: System Provided Packages¶
This will give us:
@@ -2775,85 +2816,125 @@ Installing ActiveMQsudo apt-cache policy activemq
Write down the version listed under Installed:.
+Option 2: Manual Install¶
+Git the latest ActiveMQ 5.x version number from https://archive.apache.org/dist/activemq which will be put in place of [ACTIVEMQ_VERSION_NUMBER].
+cd /opt
+sudo wget http://archive.apache.org/dist/activemq/[ACTIVEMQ_VERSION_NUMBER]/apache-activemq-[ACTIVEMQ_VERSION_NUMBER]-bin.tar.gz
+sudo tar -xvzf apache-activemq-[ACTIVEMQ_VERSION_NUMBER]-bin.tar.gz
+sudo mv apache-activemq-[ACTIVEMQ_VERSION_NUMBER] /opt/activemq
+sudo addgroup --quiet --system activemq
+sudo adduser --quiet --system --ingroup activemq --no-create-home --disabled-password activemq
+sudo chown -R activemq:activemq /opt/activemq
+sudo rm -R apache-activemq-[ACTIVEMQ_VERSION_NUMBER]-bin.tar.gz
+
+Add ActiveMQ as a service:
+`/etc/systemd/system/activemq.service | root:root/644
+
[Unit]
+Description=Apache ActiveMQ
+After=network.target
+
+[Service]
+Type=forking
+User=activemq
+Group=activemq
+
+ExecStart=/opt/activemq/bin/activemq start
+ExecStop=/opt/activemq/bin/activemq stop
+
+[Install]
+WantedBy=multi-user.target
+
+Update the WebConsolePort host property settings in /opt/activemq/conf/jetty.xml from <property name="host" value="127.0.0.1"/> to <property name="host" value="0.0.0.0"/> so that you can access the dashboard from outside the local machine.
+Optionally, change the dashboard user credentials in /opt/activemq/conf/users.properties.
+
+Security Warning
+Updating the web console port and user properties are potential security holes. It is best to restrict the host setting and create a more secure username/password combination for production.
+
+Set the service to start on machine startup and start it up:
+
sudo systemctl daemon-reload
+sudo systemctl start activemq
+sudo systemctl enable activemq
+sudo systemctl status activemq
+sudo systemctl restart activemq
+sudo apt-cache policy activemq # note version number
+
+The service should now be available at http://localhost:8161/
Installing Alpaca¶
Install Java 11+ if you haven't already.
Make a directory for Alpaca and download the latest version of Alpaca from the Maven repository. E.g.
-
mkdir /opt/alpaca
-cd /opt/alpaca
-curl -L https://repo1.maven.org/maven2/ca/islandora/alpaca/islandora-alpaca-app/2.2.0/islandora-alpaca-app-2.2.0-all.jar -o alpaca.jar
+sudo mkdir /opt/alpaca
+cd /opt/alpaca
+sudo curl -L https://repo1.maven.org/maven2/ca/islandora/alpaca/islandora-alpaca-app/2.2.0/islandora-alpaca-app-2.2.0-all.jar -o alpaca.jar
Configuration¶
Alpaca is made up of several services, each of these can be enabled or disabled individually.
Alpaca takes an external file to configure its behaviour.
Look at the example.properties file to see some example settings.
The properties are:
-# Common options
-error.maxRedeliveries=4
+
This defines how many times to retry a message before failing completely.
There are also common ActiveMQ properties to setup the connection.
-# ActiveMQ options
-jms.brokerUrl=tcp://localhost:61616
+
This defines the url to the ActiveMQ broker which you installed earlier.
-jms.username=
-jms.password=
+
This defines the login credentials (if required)
-jms.connections=10
+
This defines the pool of connections to the ActiveMQ instance.
-
-This defines how many messages to process simultaneously.
islandora-indexing-fcrepo¶
This service manages a Drupal node into a corresponding Fedora resource.
It's properties are:
-# Fcrepo indexer options
-fcrepo.indexer.enabled=true
+
This defines whether the Fedora indexer is enabled or not.
-fcrepo.indexer.node=queue:islandora-indexing-fcrepo-content
-fcrepo.indexer.delete=queue:islandora-indexing-fcrepo-delete
-fcrepo.indexer.media=queue:islandora-indexing-fcrepo-media
-fcrepo.indexer.external=queue:islandora-indexing-fcrepo-file-external
+fcrepo.indexer.node=queue:islandora-indexing-fcrepo-content
+fcrepo.indexer.delete=queue:islandora-indexing-fcrepo-delete
+fcrepo.indexer.media=queue:islandora-indexing-fcrepo-media
+fcrepo.indexer.external=queue:islandora-indexing-fcrepo-file-external
These define the various queues to listen on for the indexing/deletion
messages. The part after queue: should match your Islandora instance "Actions".
-fcrepo.indexer.milliner.baseUrl=http://localhost:8000/milliner
+
This defines the location of your Milliner microservice.
-fcrepo.indexer.concurrent-consumers=1
-fcrepo.indexer.max-concurrent-consumers=1
+
These define the default number of concurrent consumers and maximum number of concurrent
consumers working off your ActiveMQ instance.
A value of -1 means no setting is applied.
-fcrepo.indexer.async-consumer=true
+
This property allows the concurrent consumers to process concurrently; otherwise, the consumers will wait to the previous message has been processed before executing.
islandora-indexing-triplestore¶
This service indexes the Drupal node into the configured triplestore
It's properties are:
-# Triplestore indexer options
-triplestore.indexer.enabled=false
+
This defines whether the Triplestore indexer is enabled or not.
-triplestore.index.stream=queue:islandora-indexing-triplestore-index
-triplestore.delete.stream=queue:islandora-indexing-triplestore-delete
+triplestore.index.stream=queue:islandora-indexing-triplestore-index
+triplestore.delete.stream=queue:islandora-indexing-triplestore-delete
These define the various queues to listen on for the indexing/deletion
messages. The part after queue: should match your Islandora instance "Actions".
-triplestore.baseUrl=http://localhost:8080/bigdata/namespace/kb/sparql
+
This defines the location of your triplestore's SPARQL update endpoint.
-triplestore.indexer.concurrent-consumers=1
-triplestore.indexer.max-concurrent-consumers=1
+
These define the default number of concurrent consumers and maximum number of concurrent
consumers working off your ActiveMQ instance.
A value of -1 means no setting is applied.
-triplestore.indexer.async-consumer=true
+
This property allows the concurrent consumers to process concurrently; otherwise, the consumers will wait to the previous message has been processed before executing.
islandora-connector-derivative¶
@@ -2861,51 +2942,51 @@ islandora-connector-derivativederivative.<item>.enabled=true
+
This defines if the item service is enabled.
-derivative.<item>.in.stream=queue:islandora-item-connector.index
+
This is the input queue for the derivative microservice.
The part after queue: should match your Islandora instance "Actions".
-derivative.<item>.service.url=http://example.org/derivative/convert
+
This is the microservice URL to process the request.
-derivative.<item>.concurrent-consumers=1
-derivative.<item>.max-concurrent-consumers=1
+
These define the default number of concurrent consumers and maximum number of concurrent
consumers working off your ActiveMQ instance.
A value of -1 means no setting is applied.
-derivative.<item>.async-consumer=true
+
This property allows the concurrent consumers to process concurrently; otherwise, the consumers will wait to the previous message has been processed before executing.
For example, with two services defined (houdini and crayfits) my configuration would have
-derivative.systems.installed=houdini,fits
-
-derivative.houdini.enabled=true
-derivative.houdini.in.stream=queue:islandora-connector-houdini
-derivative.houdini.service.url=http://127.0.0.1:8000/houdini/convert
-derivative.houdini.concurrent-consumers=1
-derivative.houdini.max-concurrent-consumers=4
-derivative.houdini.async-consumer=true
-
-derivative.fits.enabled=true
-derivative.fits.in.stream=queue:islandora-connector-fits
-derivative.fits.service.url=http://127.0.0.1:8000/crayfits
-derivative.fits.concurrent-consumers=2
-derivative.fits.max-concurrent-consumers=2
-derivative.fits.async-consumer=false
+derivative.systems.installed=houdini,fits
+
+derivative.houdini.enabled=true
+derivative.houdini.in.stream=queue:islandora-connector-houdini
+derivative.houdini.service.url=http://127.0.0.1/houdini/convert
+derivative.houdini.concurrent-consumers=1
+derivative.houdini.max-concurrent-consumers=4
+derivative.houdini.async-consumer=true
+
+derivative.fits.enabled=true
+derivative.fits.in.stream=queue:islandora-connector-fits
+derivative.fits.service.url=http://127.0.0.1/crayfits
+derivative.fits.concurrent-consumers=2
+derivative.fits.max-concurrent-consumers=2
+derivative.fits.async-consumer=false
Customizing HTTP client timeouts¶
You can alter the HTTP client from the defaults for its request, connection and socket timeouts.
To do this you want to enable the request configurer.
-request.configurer.enabled=true
+
Then set the next 3 timeouts (measured in milliseconds) to the desired timeout.
-request.timeout=-1
-connection.timeout=-1
-socket.timeout=-1
+
The default for all three is -1 which indicates no timeout.
Alter HTTP options¶
@@ -2915,51 +2996,54 @@ Alter HTTP optionsconfiguration parameters or alter the existing defaults. You can
add them as a comma separated list of key=value pairs.
For example
-
http.additional_options=authMethod=Basic,authUsername=Jim,authPassword=1234
+
These will be added to ALL http endpoint requests.
-Note: We are currently running Camel 3.7.6, some configuration parameters on the above linked page might not be supported.
+
+Check Camel Configuration Parameters
+We are currently running Camel 3.7.6, some configuration parameters on the above linked page might not be supported.
+
Deploying/Running¶
You can see the options by passing the -h|--help flag
-> java -jar /opt/alpaca/alpaca.jar -h
-Usage: alpaca [-hV] [-c=<configurationFilePath>]
- -h, --help Show this help message and exit.
- -V, --version Print version information and exit.
- -c, --config=<configurationFilePath>
- The path to the configuration file
+> java -jar /opt/alpaca/alpaca.jar -h
+Usage: alpaca [-hV] [-c=<configurationFilePath>]
+ -h, --help Show this help message and exit.
+ -V, --version Print version information and exit.
+ -c, --config=<configurationFilePath>
+ The path to the configuration file
Using the -V|--version flag will just return the current version of the application.
-> java -jar /opt/alpaca/alpaca.jar -v
-2.0.0
+
To start Alpaca you would pass the external property file with the -c|--config flag.
For example if you are using an external properties file located at /opt/alpaca/alpaca.properties,
you would run:
-java -jar alpaca.jar -c /opt/alpaca/alpaca.properties
+
To use systemd to start and stop the service create the file /etc/systemd/system/alpaca.service:
-[Unit]
-Description=Alpaca service
-After=network.target
-
-[Service]
-Type=forking
-ExecStart=java -jar /opt/alpaca/alpaca.jar -c /opt/alpaca/alpaca.properties
-ExecStop=/bin/kill -15 $MAINPID
-SuccessExitStatus=143
-Restart=always
-
-[Install]
-WantedBy=default.target
+[Unit]
+Description=Alpaca service
+After=network.target
+
+[Service]
+Type=simple
+ExecStart=java -jar /opt/alpaca/alpaca.jar -c /opt/alpaca/alpaca.properties
+ExecStop=/bin/kill -15 $MAINPID
+SuccessExitStatus=143
+Restart=always
+
+[Install]
+WantedBy=default.target
-Now you can start the service by running systemctl start alpaca and set it to come up when the system reboots with systemctl enable alpaca.
+Now you can start the service by running sudo systemctl start alpaca and set it to come up when the system reboots with sudo systemctl enable alpaca. Check the status by running sudo systemctl status alpaca.
Last update:
- July 15, 2024
+ August 16, 2024
diff --git a/pr-preview/pr-2331/installation/manual/installing-composer-drush-and-drupal/index.html b/pr-preview/pr-2331/installation/manual/installing-composer-drush-and-drupal/index.html
index 8b7d1ff6b..959f08252 100644
--- a/pr-preview/pr-2331/installation/manual/installing-composer-drush-and-drupal/index.html
+++ b/pr-preview/pr-2331/installation/manual/installing-composer-drush-and-drupal/index.html
@@ -496,7 +496,7 @@
-
+
8.x-2.0 Release Notes
@@ -1020,6 +1020,13 @@
In this section, we will install:
+
+
+
+
+ Install cURL
+
+
@@ -1077,8 +1084,8 @@
-
- Prepare the PostgreSQL database
+
+ Prepare the MySQL database
@@ -2661,6 +2668,13 @@
In this section, we will install:
+
+
+
+
+ Install cURL
+
+
@@ -2718,8 +2732,8 @@
-
- Prepare the PostgreSQL database
+
+ Prepare the MySQL database
@@ -2778,42 +2792,50 @@
Installing Composer, Drush, and Drupal¶
Needs Maintenance
-The manual installation documentation is in need of attention. We are aware that some components no longer work as documented here. If you are interested in helping us improve the documentation, please see Contributing.
+The manual installation documentation is in need of attention. We are aware that some components no longer work as documented here. If you are interested in helping us improve the documentation, please see Contributing.
In this section, we will install:¶
+- cURL is used by composer to efficiently download libraries
- Composer at its current latest version, the package manager that will allow us to install PHP applications
- Either the Islandora Starter Site, or the Drupal recommended-project, which will install, among other things:
-- Drush 10 at its latest version, the command-line PHP application for running tasks in Drupal
-- Drupal 9 at its latest version, the content management system Islandora uses for content modelling and front-end display
+- Drush at its latest version, the command-line PHP application for running tasks in Drupal
+- Drupal at its latest version, the content management system Islandora uses for content modelling and front-end display
+Install cURL¶
+cURL may already be installed. Check by running curl --version. If it isn't, install it:
+
Install Composer¶
Download and install Composer 2.x¶
Composer provides PHP code that we can use to install it. After downloading and running the installer, we’re going to move the generated executable to a place in $PATH, removing its extension:
-curl "https://getcomposer.org/installer" > composer-install.php
-chmod +x composer-install.php
-php composer-install.php
-sudo mv composer.phar /usr/local/bin/composer
+curl "https://getcomposer.org/installer" > composer-install.php
+chmod +x composer-install.php
+php composer-install.php
+sudo mv composer.phar /usr/local/bin/composer
Download and Scaffold Drupal¶
At this point, you have the option of using the Islandora Starter Site, with its pre-selected modules
and configurations which function "out of the box," or build a clean stock Drupal via the Drupal Recommended Project and install
Islandora modules as you desire.
+On a default Ubuntu install the /var/www directory is owned by root, but we want the webserver to control this space, so we'll give it ownership:
+
Option 1: Create a project using the Islandora Starter Site¶
-Navigate to the folder where you want to put your Islandora project (in our case /var/www), and
-create the Islandora Starter Site:
-cd /var/www
-composer create-project islandora/islandora-starter-site
+Navigate to the folder where you want to put your Islandora project (in our case /var/www/html). You can give your site any name you like, but for these instructions we will simply call it "drupal":
+cd /var/www/html
+sudo -u www-data mkdir drupal
+sudo -u www-data composer create-project islandora/islandora-starter-site drupal
This will install all PHP dependencies, including Drush, and scaffold the site.
Drush is not accessible via $PATH, but is available using the command composer exec -- drush
Option 2: Create a basic Drupal Recommended Project¶
-Navigate to the folder where you want to put your Drupal project (in our case /var/www), and
+
Navigate to the folder where you want to put your Drupal project (in our case /var/www/html), and
create the Drupal Recommended Project:
-cd /var/www
-composer create-project drupal/recommended-project my-project
+
Make the new webroot accessible in Apache¶
Before we can proceed with the actual site installation, we’re going to need to make our new Drupal installation the default web-accessible location Apache serves up. This will include an appropriate ports.conf file, and replacing the default enabled site.
@@ -2822,54 +2844,58 @@ Make the new webroot accessib
Out of the box, these files will contain support for SSL, which we will not be setting up in this guide (and therefore removing with these overwritten configurations), but which are absolutely indispensable to a production site. This guide does not recommend any particular SSL certificate authority or installation method, but you may find DigitalOcean's tutorial helpful.
/etc/apache2/ports.conf | root:root/644
-
Listen 80
+
Remove everything but the "Listen 80" line. You can leave the comments in if you want.
-/etc/apache2/sites-enabled/000-default.conf | root:root/777
-
<VirtualHost *:80>
- ServerName SERVER_NAME
- DocumentRoot "/opt/drupal/web"
- <Directory "/opt/drupal/web">
- Options Indexes FollowSymLinks MultiViews
- AllowOverride all
- Require all granted
- </Directory>
- # Ensure some logging is in place.
- ErrorLog "/var/log/apache2/localhost_error.log"
- CustomLog "/var/log/apache2/localhost_access.log" combined
-</VirtualHost>
+Create a drupal virtual host:
+/etc/apache2/sites-available/islandora.conf | root:root/644
+
<VirtualHost *:80>
+ ServerName SERVER_NAME
+ DocumentRoot "/var/www/html/drupal/web"
+ <Directory "/var/www/html/drupal/web">
+ Options Indexes FollowSymLinks MultiViews
+ AllowOverride all
+ Require all granted
+ </Directory>
+ # Ensure some logging is in place.
+ ErrorLog "/var/log/apache2/localhost_error.log"
+ CustomLog "/var/log/apache2/localhost_access.log" combined
+</VirtualHost>
- SERVER_NAME: localhost
- For a development environment hosted on your own machine or a VM, localhost should suffice. Realistically, this should be the domain or IP address the server will be accessed at.
-Restart the Apache 2 service to apply these changes:
-sudo systemctl restart apache2
+Set permissions and enable the virtual host:
+
-Prepare the PostgreSQL database¶
-PostgreSQL roles are directly tied to users. We’re going to ensure a user is in place, create a role for them in PostgreSQL, and create a database for them that we can use to install Drupal.
-# Run psql as the postgres user, the only user currently with any PostgreSQL
-# access.
-sudo -u postgres psql
-# Then, run these commands within psql itself:
-create database DRUPAL_DB encoding 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' TEMPLATE template0;
-create user DRUPAL_DB_USER with encrypted password 'DRUPAL_DB_PASSWORD';
-grant all privileges on database DRUPAL_DB to DRUPAL_DB_USER;
-# Then, quit psql.
-\q
+Prepare the MySQL database¶
+We're going to create a user in MySQL for this Drupal site. Then create a database that we can use to install Drupal.
+The following values can (and in the case of the password, should) be changed to local values.
+
+DRUPAL_DATABASE_NAME: This will be used as the core database that Drupal is installed intoMYSQL_USER_FOR_DRUPAL: Specifically, this is the user that will connect to the MySQL database being created, not the user that will be logging into DrupalMYSQL_PASSWORD_FOR_DRUPAL: This should be a secure password; it’s recommended to use a password generator to create this such as the one provided by random.org
+sudo mysql -u root
+CREATE DATABASE [DRUPAL_DATABASE_NAME];
+CREATE USER '[MYSQL_USER_FOR_DRUPAL]'@'localhost' IDENTIFIED BY '[MYSQL_PASSWORD_FOR_DRUPAL]';
+GRANT ALL PRIVILEGES ON [DRUPAL_DATABASE_NAME].* TO '[MYSQL_USER_FOR_DRUPAL]'@'localhost';
+exit
-- DRUPAL_DB: drupal9
- - This will be used as the core database that Drupal is installed into
-- DRUPAL_DB_USER: drupal
- - Specifically, this is the user that will connect to the PostgreSQL database being created, not the user that will be logging into Drupal
-- DRUPAL_DB_PASSWORD: drupal
- - This should be a secure password; it’s recommended to use a password generator to create this such as the one provided by random.org
Install Drupal using Drush¶
-The Drupal installation process can be done through the GUI in a series of form steps, or can be done quickly using Drush's site-install command. It can be invoked with the full list of parameters (such as --db-url and --site-name), but if parameters are missing, they will be asked of you interactively.
+The Drupal installation process can be done through the GUI in a series of form steps, or can be done quickly using Drush's site-install command. It can be invoked with the full list of parameters (such as --db-url and --site-name), but if parameters are missing, they will be asked of you interactively.
Option 1: Site install the Starter Site with existing configs¶
Follow the instructions in the README of the Islandora Starter Site.
-The steps are not reproduced here to remove redundancy. When this installation is done, you'll have a starter site ready-to-go. Once you set up the external services in the next sections, you'll need to configure Drupal to know where they are.
+The steps are not reproduced here to remove redundancy. But specifically,
+
+- Configure the database connection information (see the section above) and fedora flysystem in
/var/www/html/drupal/web/sites/default/settings.php.
+- Install the site using
sudo -u www-data composer exec -- drush site:install --existing-config.
+
+When this installation is done, you'll have a starter site ready-to-go. Once you set up the external services in the next sections, you'll need to configure Drupal to know where they are.
Option 2: Site install the basic Drupal Recommended Project¶
-cd /var/www/drupal
-drush -y site-install standard --db-url="pgsql://DRUPAL_DB_USER:DRUPAL_DB_PASSWORD@127.0.0.1:5432/DRUPAL_DB" --site-name="SITE_NAME" --account-name=DRUPAL_LOGIN --account-pass=DRUPAL_PASS
+cd /var/www/drupal
+sudo -u www-data drush -y site-install standard --db-url="mysql://MYSQL_USER_FOR_DRUPAL:MYSQL_PASSWORD_FOR_DRUPAL@127.0.0.1:3306/DRUPAL_DATABASE_NAME" --site-name="SITE_NAME" --account-name=DRUPAL_LOGIN --account-pass=DRUPAL_PASS
This uses the same parameters from the above step, as well as:
@@ -2893,7 +2919,7 @@ Option 2: Si
Last update:
- July 15, 2024
+ August 16, 2024
diff --git a/pr-preview/pr-2331/installation/manual/installing-crayfish/index.html b/pr-preview/pr-2331/installation/manual/installing-crayfish/index.html
index 02d71afee..3822c3ad5 100644
--- a/pr-preview/pr-2331/installation/manual/installing-crayfish/index.html
+++ b/pr-preview/pr-2331/installation/manual/installing-crayfish/index.html
@@ -496,7 +496,7 @@
-
-
+
8.x-2.0 Release Notes
@@ -1076,6 +1076,13 @@
In this section, we will install:
+
+
+ -
+
+ FITS Web Service
+
+
-
@@ -1141,13 +1148,6 @@
Milliner (Fedora indexing)
-
-
- -
-
- Recast (Drupal to Fedora URI re-writing)
-
-
@@ -2676,6 +2676,13 @@
In this section, we will install:
+
+
+
+
+ FITS Web Service
+
+
@@ -2741,13 +2748,6 @@
Milliner (Fedora indexing)
-
-
-
-
- Recast (Drupal to Fedora URI re-writing)
-
-
@@ -2808,13 +2808,36 @@
Installing Crayfish¶
Needs Maintenance
-The manual installation documentation is in need of attention. We are aware that some components no longer work as documented here. If you are interested in helping us improve the documentation, please see Contributing.
+The manual installation documentation is in need of attention. We are aware that some components no longer work as documented here. If you are interested in helping us improve the documentation, please see Contributing.
In this section, we will install:¶
+- FITS Web Service, a webservice for identifying file metadata
- Islandora/Crayfish, the suite of microservices that power the backend of Islandora 2.0
- Indvidual microservices underneath Crayfish
+FITS Web Service¶
+The FITS Web Service is used to extract file metadata from files. The Crayfish microservice CrayFits will use this service to push FITS metadata back to Drupal. It comes in two pieces, the actual FITS tool and the FITS Webservice which runs in Tomcat.
+FITS itself wraps other file identification and metadata tools which may require installing additional libraries. On Ububtu 20.04, the version this guide is using, we will install a few:
+
+To set up the FITS application, first find the latest FITS version on GitHub to replace the [FITS_VERSION_NUMBER] and then run the following commands:
+cd /opt
+sudo wget https://github.com/harvard-lts/fits/releases/download/[FITS_VERSION_NUMBER]/fits-[FITS_VERSION_NUMBER].zip
+sudo unzip /opt/fits-[FITS_VERSION_NUMBER].zip -d /opt/fits
+
+Similarly with the FITS webservice, get the current service version number to replace [FITS_SERVICE_WAR_VERSION_NUMBER]:
+Download the FITS webservice:
+sudo -u tomcat wget -O /opt/tomcat/webapps/fits.war https://github.com/harvard-lts/FITSservlet/releases/download/[FITS_SERVICE_WAR_VERSION_NUMBER]/fits-service-[FITS_SERVICE_WAR_VERSION_NUMBER].war
+
+Configure the webservice but adding the following lines to the bottom of /opt/tomcat/conf/catalina.properties:
+
+Restart Tomcat:
+
+Wait for a few minutes to let the service start up the first time and then visit http://localhost:8080/fits/ to ensure it is working. You can also follow the catalina logs to see how tomcat is progressing in setting up each service it is running: sudo tail -f /opt/tomcat/logs/catalina.out. To stop following the logs, hit control-C.
Crayfish 2.0¶
Installing Prerequisites¶
Some packages need to be installed before we can proceed with installing Crayfish; these packages are used by the microservices within Crayfish. These include:
@@ -2824,296 +2847,246 @@ Installing PrerequisitesFFMPEG, which will be used for video processing
Poppler, which will be used for generating PDFs
-sudo add-apt-repository -y ppa:lyrasis/imagemagick-jp2
-sudo apt-get update
-sudo apt-get -y install imagemagick tesseract-ocr ffmpeg poppler-utils
+sudo apt-get install software-properties-common
+sudo add-apt-repository -y ppa:lyrasis/imagemagick-jp2
+sudo apt-get update
+sudo apt-get -y install imagemagick tesseract-ocr ffmpeg poppler-utils
-NOTICE: If you get the sudo: apt-add-repository: command not found, run sudo apt-get install software-properties-common in order to make the command available.
Cloning and Installing Crayfish¶
We’re going to clone Crayfish to /opt, and individually run composer install against each of the microservice subdirectories.
-cd /opt
-sudo git clone https://github.com/Islandora/Crayfish.git crayfish
-sudo chown -R www-data:www-data crayfish
-sudo -u www-data composer install -d crayfish/Homarus
-sudo -u www-data composer install -d crayfish/Houdini
-sudo -u www-data composer install -d crayfish/Hypercube
-sudo -u www-data composer install -d crayfish/Milliner
-sudo -u www-data composer install -d crayfish/Recast
+cd /opt
+sudo git clone https://github.com/Islandora/Crayfish.git crayfish
+sudo chown -R www-data:www-data crayfish
+sudo -u www-data composer install -d crayfish/Homarus
+sudo -u www-data composer install -d crayfish/Houdini
+sudo -u www-data composer install -d crayfish/Hypercube
+sudo -u www-data composer install -d crayfish/Milliner
+sudo -u www-data composer install -d crayfish/Recast
+sudo -u www-data composer install -d crayfish/CrayFits
Preparing Logging¶
Not much needs to happen here; Crayfish opts for a simple logging approach, with one .log file for each component. We’ll create a folder where each logfile can live.
-sudo mkdir /var/log/islandora
-sudo chown www-data:www-data /var/log/islandora
+
Configuring Crayfish Components¶
Each Crayfish component requires one or more .yaml file(s) to ensure everything is wired up correctly.
-NOTICE
+
+Update the defaults to meet your needs
+
The following configuration files represent somewhat sensible defaults; you should take consideration of the logging levels in use, as this can vary in desirability from installation to installation. Also note that in all cases, http URLs are being used, as this guide does not deal with setting up https support. In a production installation, this should not be the case. These files also assume a connection to a PostgreSQL database; use a pdo_mysql driver and the appropriate 3306 port if using MySQL.
+
+Using JWT for Crayfish Authentication
+For Crayfish microservices use the lexik_jwt_authentication package. They are configured to use the JWT_PUBLIC_KEY environment variable to find the public key we created earlier (/opt/keys/syn_public.key). Later on in this guide we will add the environment variable to the Apache configs, but you may alternatively write the path to the key in the lexik_jwt_authentication.yaml file that resides along-side the security.yaml files we edit in this section.
+
Homarus (Audio/Video derivatives)¶
-/opt/crayfish/Homarus/cfg/config.yaml | www-data:www-data/644
-
---
-homarus:
- executable: ffmpeg
- mime_types:
- valid:
- - video/mp4
- - video/x-msvideo
- - video/ogg
- - audio/x-wav
- - audio/mpeg
- - audio/aac
- - image/jpeg
- - image/png
- default: video/mp4
- mime_to_format:
- valid:
- - video/mp4_mp4
- - video/x-msvideo_avi
- - video/ogg_ogg
- - audio/x-wav_wav
- - audio/mpeg_mp3
- - audio/aac_m4a
- - image/jpeg_image2pipe
- - image/png_image2pipe
- default: mp4
-fedora_resource:
- base_url: http://localhost:8080/fcrepo/rest
-log:
- level: NOTICE
- file: /var/log/islandora/homarus.log
-syn:
- enable: true
- config: /opt/fcrepo/config/syn-settings.xml
-
+Enable JSON Web Token (JWT) based access to the service by updating the security settings. Edit /opt/crayfish/Homarus/config/packages/security.yaml to set firewalls: main: anonymous to false and uncomment the provider and jwt lines further down in that section.
+Edit /opt/crayfish/Homarus/config/packages/monolog.yaml to point to the new logging directory:
+
+Edit the commons config to update it with Fedora's location (if necessary) and enable the apix middleware in /opt/crayfish/Homarus/config/packages/crayfish_commons.yaml:
+crayfish_commons:
+ fedora_base_uri: 'http://localhost:8080/fcrepo/rest'
+ apix_middleware_enabled: true
+
Houdini (Image derivatives)¶
Currently the Houdini microservice uses a different system (Symfony) than the other microservices, this requires different configuration.
/opt/crayfish/Houdini/config/services.yaml | www-data:www-data/644
-
# This file is the entry point to configure your own services.
-# Files in the packages/ subdirectory configure your dependencies.
-# Put parameters here that don't need to change on each machine where the app is deployed
-# https://symfony.com/doc/current/best_practices/configuration.html#application-related-configuration
-parameters:
- app.executable: /usr/local/bin/convert
- app.formats.valid:
- - image/jpeg
- - image/png
- - image/tiff
- - image/jp2
- app.formats.default: image/jpeg
-
-services:
- # default configuration for services in *this* file
- _defaults:
- autowire: true # Automatically injects dependencies in your services.
- autoconfigure: true # Automatically registers your services as commands, event subscribers, etc.
-
- # makes classes in src/ available to be used as services
- # this creates a service per class whose id is the fully-qualified class name
- App\Islandora\Houdini\:
- resource: '../src/*'
- exclude: '../src/{DependencyInjection,Entity,Migrations,Tests,Kernel.php}'
-
- # controllers are imported separately to make sure services can be injected
- # as action arguments even if you don't extend any base controller class
- App\Islandora\Houdini\Controller\HoudiniController:
- public: false
- bind:
- $formats: '%app.formats.valid%'
- $default_format: '%app.formats.default%'
- $executable: '%app.executable%'
- tags: ['controller.service_arguments']
-
- # add more service definitions when explicit configuration is needed
- # please note that last definitions always *replace* previous ones
+# This file is the entry point to configure your own services.
+# Files in the packages/ subdirectory configure your dependencies.
+# Put parameters here that don't need to change on each machine where the app is deployed
+# https://symfony.com/doc/current/best_practices/configuration.html#application-related-configuration
+parameters:
+ app.executable: /usr/bin/convert
+ app.formats.valid:
+ - image/jpeg
+ - image/png
+ - image/tiff
+ - image/jp2
+ app.formats.default: image/jpeg
+
+services:
+ # default configuration for services in *this* file
+ _defaults:
+ autowire: true # Automatically injects dependencies in your services.
+ autoconfigure: true # Automatically registers your services as commands, event subscribers, etc.
+
+ # makes classes in src/ available to be used as services
+ # this creates a service per class whose id is the fully-qualified class name
+ App\Islandora\Houdini\:
+ resource: '../src/*'
+ exclude: '../src/{DependencyInjection,Entity,Migrations,Tests,Kernel.php}'
+
+ # controllers are imported separately to make sure services can be injected
+ # as action arguments even if you don't extend any base controller class
+ App\Islandora\Houdini\Controller\HoudiniController:
+ public: false
+ bind:
+ $formats: '%app.formats.valid%'
+ $default_format: '%app.formats.default%'
+ $executable: '%app.executable%'
+ tags: ['controller.service_arguments']
+
+ # add more service definitions when explicit configuration is needed
+ # please note that last definitions always *replace* previous ones
-/opt/crayfish/Houdini/config/packages/crayfish_commons.yml | www-data:www-data/644
-
crayfish_commons:
- fedora_base_uri: 'http://localhost:8080/fcrepo/rest'
- syn_config: '/opt/fcrepo/config/syn-settings.xml'
+/opt/crayfish/Houdini/config/packages/crayfish_commons.yaml | www-data:www-data/644
+
crayfish_commons:
+ fedora_base_uri: 'http://localhost:8080/fcrepo/rest'
+ syn_config: /opt/fcrepo/config/syn-settings.xml
+ syn_enabled: True
-/opt/crayfish/Houdini/config/packages/monolog.yml | www-data:www-data/644
-
monolog:
-
- handlers:
-
- houdini:
- type: rotating_file
- path: /var/log/islandora/Houdini.log
- level: DEBUG
- max_files: 1
+/opt/crayfish/Houdini/config/packages/monolog.yaml | www-data:www-data/644
+
monolog:
+
+ handlers:
+
+ houdini:
+ type: rotating_file
+ path: /var/log/islandora/Houdini.log
+ level: DEBUG
+ max_files: 1
The below files are two versions of the same file to enable or disable JWT token authentication.
-/opt/crayfish/Houdini/config/packages/security.yml | www-data:www-data/644
+/opt/crayfish/Houdini/config/packages/security.yaml | www-data:www-data/644
Enabled JWT token authentication:
-
security:
-
- # https://symfony.com/doc/current/security.html#where-do-users-come-from-user-providers
- providers:
- jwt_user_provider:
- id: Islandora\Crayfish\Commons\Syn\JwtUserProvider
-
- firewalls:
- dev:
- pattern: ^/(_(profiler|wdt)|css|images|js)/
- security: false
- main:
- anonymous: false
- # Need stateless or it reloads the User based on a token.
- stateless: true
-
- provider: jwt_user_provider
- guard:
- authenticators:
- - Islandora\Crayfish\Commons\Syn\JwtAuthenticator
-
- # activate different ways to authenticate
- # https://symfony.com/doc/current/security.html#firewalls-authentication
-
- # https://symfony.com/doc/current/security/impersonating_user.html
- # switch_user: true
-
-
- # Easy way to control access for large sections of your site
- # Note: Only the *first* access control that matches will be used
- access_control:
- # - { path: ^/admin, roles: ROLE_ADMIN }
- # - { path: ^/profile, roles: ROLE_USER }
+# To disable Syn checking, set syn_enabled=false in crayfish_commons.yaml and remove this configuration file.
+security:
+
+ # https://symfony.com/doc/current/security.html#where-do-users-come-from-user-providers
+ providers:
+ users_in_memory: { memory: null }
+ jwt:
+ lexik_jwt: ~
+ firewalls:
+ dev:
+ pattern: ^/(_(profiler|wdt)|css|images|js)/
+ security: false
+ main:
+ # To enable Syn, change anonymous to false and uncomment the lines further below
+ anonymous: false
+ # Need stateless or it reloads the User based on a token.
+ stateless: true
+
+ # To enable JWT authentication, uncomment the below 2 lines and change anonymous to false above.
+ provider: jwt
+ jwt: ~
+
+ # activate different ways to authenticate
+ # https://symfony.com/doc/5.4/security.html#firewalls-authentication
+
+ # https://symfony.com/doc/5.4/security/impersonating_user.html
+ # switch_user: true
+
+ # Easy way to control access for large sections of your site
+ # Note: Only the *first* access control that matches will be used
+ access_control:
+ # - { path: ^/admin, roles: ROLE_ADMIN }
+ # - { path: ^/profile, roles: ROLE_USER }
Disabled JWT token authentication:
-
security:
-
- # https://symfony.com/doc/current/security.html#where-do-users-come-from-user-providers
- providers:
- jwt_user_provider:
- id: Islandora\Crayfish\Commons\Syn\JwtUserProvider
-
- firewalls:
- dev:
- pattern: ^/(_(profiler|wdt)|css|images|js)/
- security: false
- main:
- anonymous: true
- # Need stateless or it reloads the User based on a token.
- stateless: true
+security:
+
+ # https://symfony.com/doc/current/security.html#where-do-users-come-from-user-providers
+ providers:
+ jwt_user_provider:
+ id: Islandora\Crayfish\Commons\Syn\JwtUserProvider
+
+ firewalls:
+ dev:
+ pattern: ^/(_(profiler|wdt)|css|images|js)/
+ security: false
+ main:
+ anonymous: true
+ # Need stateless or it reloads the User based on a token.
+ stateless: true
Hypercube (OCR)¶
-/opt/crayfish/Hypercube/cfg/config.yaml | www-data:www-data/644
-
---
-hypercube:
- tesseract_executable: tesseract
- pdftotext_executable: pdftotext
-fedora_resource:
- base_url: http://localhost:8080/fcrepo/rest
-log:
- level: NOTICE
- file: /var/log/islandora/hypercube.log
-syn:
- enable: true
- config: /opt/fcrepo/config/syn-settings.xml
-
+Enable JSON Web Token (JWT) based access to the service by updating the security settings. Edit /opt/crayfish/Hypercube/config/packages/security.yaml to set firewalls: main: anonymous to false and uncomment the provider and jwt lines further down in that section.
+Edit /opt/crayfish/Hypercube/config/packages/monolog.yaml to point to the new logging directory:
+
+Edit the commons config to update it with Fedora's location (if necessary) and enable the apix middleware in /opt/crayfish/Hypercube/config/packages/crayfish_commons.yaml:
+crayfish_commons:
+ fedora_base_uri: 'http://localhost:8080/fcrepo/rest'
+ apix_middleware_enabled: true
+
Milliner (Fedora indexing)¶
-/opt/crayfish/Milliner/cfg/config.yaml | www-data:www-data/644
-
---
-fedora_base_url: http://localhost:8080/fcrepo/rest
-drupal_base_url: http://localhost
-modified_date_predicate: http://schema.org/dateModified
-strip_format_jsonld: true
-debug: false
-db.options:
- driver: pdo_pgsql
- host: 127.0.0.1
- port: 5432
- dbname: CRAYFISH_DB
- user: CRAYFISH_DB_USER
- password: CRAYFISH_DB_PASSWORD
-log:
- level: NOTICE
- file: /var/log/islandora/milliner.log
-syn:
- enable: true
- config: /opt/fcrepo/config/syn-settings.xml
-
-Recast (Drupal to Fedora URI re-writing)¶
-/opt/crayfish/Recast/cfg/config.yaml | www-data:www-data/644
-
---
-fedora_resource:
- base_url: http://localhost:8080/fcrepo/rest
-drupal_base_url: http://localhost
-debug: false
-log:
- level: NOTICE
- file: /var/log/islandora/recast.log
-syn:
- enable: true
- config: /opt/fcrepo/config/syn-settings.xml
-namespaces:
--
- acl: "http://www.w3.org/ns/auth/acl#"
- fedora: "http://fedora.info/definitions/v4/repository#"
- ldp: "http://www.w3.org/ns/ldp#"
- memento: "http://mementoweb.org/ns#"
- pcdm: "http://pcdm.org/models#"
- pcdmuse: "http://pcdm.org/use#"
- webac: "http://fedora.info/definitions/v4/webac#"
- vcard: "http://www.w3.org/2006/vcard/ns#"
-
+Enable JSON Web Token (JWT) based access to the service by updating the security settings. Edit /opt/crayfish/Milliner/config/packages/security.yaml to set firewalls: main: anonymous to false and uncomment the provider and jwt lines further down in that section.
+Edit /opt/crayfish/Milliner/config/packages/monolog.yaml to point to the new logging directory:
+
+Edit the commons config to update it with Fedora's location (if necessary) and enable the apix middleware in /opt/crayfish/Milliner/config/packages/crayfish_commons.yaml:
Creating Apache Configurations for Crayfish Components¶
Finally, we need appropriate Apache configurations for Crayfish; these will allow other services to connect to Crayfish components via their HTTP endpoints.
Each endpoint we need to be able to connect to will get its own .conf file, which we will then enable.
-NOTICE
+
+Possible Route Collisions
These configurations would potentially have collisions with Drupal routes, if any are created in Drupal with the same name. If this is a concern, it would likely be better to reserve a subdomain or another port specifically for Crayfish. For the purposes of this installation guide, these endpoints will suffice.
+
/etc/apache2/conf-available/Homarus.conf | root:root/644
-
Alias "/homarus" "/opt/crayfish/Homarus/src"
-<Directory "/opt/crayfish/Homarus/src">
- FallbackResource /homarus/index.php
- Require all granted
- DirectoryIndex index.php
- SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
-</Directory>
+Alias "/homarus" "/opt/crayfish/Homarus/public"
+<Directory "/opt/crayfish/Homarus/public">
+ FallbackResource /homarus/index.php
+ Require all granted
+ DirectoryIndex index.php
+ SetEnv JWT_PUBLIC_KEY /opt/keys/syn_public.key
+ SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
+</Directory>
/etc/apache2/conf-available/Houdini.conf | root:root/644
-
Alias "/houdini" "/opt/crayfish/Houdini/public"
-<Directory "/opt/crayfish/Houdini/public">
- FallbackResource /houdini/index.php
- Require all granted
- DirectoryIndex index.php
- SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
-</Directory>
+Alias "/houdini" "/opt/crayfish/Houdini/public"
+<Directory "/opt/crayfish/Houdini/public">
+ FallbackResource /houdini/index.php
+ Require all granted
+ DirectoryIndex index.php
+ SetEnv JWT_PUBLIC_KEY /opt/keys/syn_public.key
+ SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
+</Directory>
/etc/apache2/conf-available/Hypercube.conf | root:root/644
-