Simplify sandbox, and rename to demo (#55)

* Fix logging for microservices, now goes to stderr.

* Refactored sandbox/drupal to make customizing drupal images easier.

* Hopefully fix address resolution issues with Github Actions

.gitattributes

@@ -12,9 +12,13 @@ gradlew text eol=lf diff=bash
 *.patch text eol=lf diff=batch
 *.properties text eol=lf
 *.sh text eol=lf diff=bash
+*.toml text eol=lf diff=bash
+*.tmpl text eol=lf diff=bash
 APKBUILD text eol=lf diff=batch
 finish text eol=lf diff=bash
 run text eol=lf diff=bash
+drush text eol=lf diff=bash
+composer text eol=lf diff=bash
 
 # These files are binary and should be left untouched
 # (binary is a macro for -text -diff)

.github/workflows/master.yml

@@ -27,7 +27,7 @@ jobs:
     - name: Enable buildkit
       shell: bash
       run: |
-        echo '{"experimental": "enabled"}' > ~/.docker/config.json
+        echo '{"experimental": "enabled", "dns": ["8.8.8.8", "8.8.4.4"]}' > ~/.docker/config.json
     - name: Build Docker images
       uses: eskatos/gradle-command-action@v1
       with:

.gitignore

@@ -4,3 +4,4 @@
 scratch
 scratch.md
 volumes
+docker-compose.yml

.vscode/settings.json

@@ -9,6 +9,8 @@
         "*.properties.tmpl": "properties",
         "*.sh.tmpl": "shellscript",
         "*.sql.tmpl": "sql",
-        "*.xml.tmpl": "xml"
+        "*.xml.tmpl": "xml",
+        "drush": "shellscript",
+        "composer": "shellscript"
     }
 }

README.md

@@ -11,13 +11,7 @@
   - [Build Specific Image](#build-specific-image)
   - [Building Continuously](#building-continuously)
 - [Running](#running)
-- [Scripts](#scripts)
 - [Docker Images](#docker-images)
-- [Docker Compose](#docker-compose)
-  - [Watchtower](#watchtower)
-  - [Traefik](#traefik)
-- [ETCD](#etcd)
-- [Customizing the Drupal Installation](#customizing-the-drupal-installation)
 - [Design Considerations](#design-considerations)
   - [Confd](#confd)
   - [S6 Overlay](#s6-overlay)
@@ -26,13 +20,11 @@
   - [Build System](#build-system)
 - [Design Constraints](#design-constraints)
 - [Issues / FAQ](#issues--faq)
-- [To Do](#to-do)
 
 ## Introduction
 
-This repository provides a number of docker images and an example
-[docker-compose.yml](./docker-compose.yml) for running the demo version of
-Islandora. It is not yet in a production ready state.
+This repository provides a number of docker images which can be used to build an
+Islandora 8 site.
 
 ## Requirements
 
@@ -41,14 +33,8 @@ To build the Docker images using the provided Gradle build scripts with [BuildKi
 - [Docker 18.09+](https://docs.docker.com/get-docker/)
 - [OpenJDK or Oracle JDK 8+](https://www.java.com/en/download/)
 
-To run the Docker images with Docker Compose requires:
-
-- [Docker 18.06+](https://docs.docker.com/get-docker/)
-- [Docker Compose 1.22+](https://docs.docker.com/compose/install/)
-
-That being said the images themselves are compatible with older versions of
-Docker, if you require running on an older version you'll need to write your own
-docker-compose file.
+That being said the images themselves are compatible of running with older
+versions of Docker.
 
 ## Building
 
@@ -158,55 +144,13 @@ of that image and any downstream dependencies. To do this add the
 ```
 
 When this is combined with the use of ``watchtower`` and
-``restart: unless-stopped`` in the [docker-compose.yml](./docker-compose.yml)
-file. Images will be redeployed with the latest changes while you develop
-automatically. See the [Docker Compose](#Docker-Compose) section of this
-document for more details.
+``restart: unless-stopped`` in a ``docker-compose.yml`` file. Images will be
+redeployed with the latest changes while you develop automatically.
 
 ## Running
 
-At the moment the example [docker-compose.yml] is the only orchestration
-mechanism provided to launch all the containers, and have them work as a whole.
-
-To start the containers use the following command:
-
-```bash
-docker-compose up -d
-```
-
-With [Docker Compose] there are many features such as displaying logs among
-other things for which you can find detailed descriptions in the
-[Docker Composer CLI Documentation](https://docs.docker.com/compose/reference/overview/)
-
-For more information on the structure and design of the example
-[docker-compose.yml] file see the [Docker Compose](#Docker-Compose) section of
-this document.
-
-## Scripts
-
-Some helper scripts are provided to make development and testing more pleasurable.
-
-- [./commands/drush.sh](./commands/drush.sh) - Wrapper around [drush] in the ``drupal service`` container.
-- [./commands/etcdctrl.sh](./commands/etcdctrl.sh) - Wrapper around [etcdctrl] in the ``etcd service`` container.
-- [./commands/mysql.sh](./commands/mysql.sh) - Wrapper around [mysql] client in the ``database service`` container.
-- [./commands/open-in-browser.sh](./commands/shell.sh) - Attempts to open the given service in the users browser.
-- [./commands/shell.sh](./commands/shell.sh) - Open ``ash`` shell in the given service container.
-
-All of the above commands include a usage statement, which can be accessed with ``-h`` flag like so:
-
-```bash
-$ ./commands/shell.sh
-    usage: shell.sh SERVICE
-
-    Opens an ash shell in the given SERVICE's container.
-
-    OPTIONS:
-       -h --help          Show this help.
-       -x --debug         Debug this script.
-
-    Examples:
-       shell.sh database
-```
+There is no method for running the containers in `isle-buildkit`, instead please
+refer to <https://github.com/Islandora-Devops/isle-dc>.
 
 ## Docker Images
 
@@ -219,7 +163,6 @@ The following docker images are provided:
 - [blazegraph](./blazegraph/README.md)
 - [build](./build/README.md)
 - [cantaloupe](./cantaloupe/README.md)
-- [composer](./composer/README.md)
 - [crayfish](./crayfish/README.md)
 - [crayfits](./crayfits/README.md)
 - [drupal](./drupal/README.md)
@@ -237,143 +180,14 @@ The following docker images are provided:
 - [milliner](./milliner/README.md)
 - [nginx](./nginx/README.md)
 - [recast](./recast/README.md)
-- [sandbox](./sandbox/README.md)
+- [demo](./demo/README.md)
 - [solr](./solr/README.md)
 - [tomcat](./tomcat/README.md)
 
 Many are intermediate images used to build other images in the list, for example
 [java](./java/README.md). Please see the README of each image to find out what
 settings, and ports, are exposed and what functionality it provides.
 
-## Docker Compose
-
-The example [docker-compose.yml] provided with this repository is a
-template for those who wish to use [Docker Compose] for orchestration. The
-images are not limited to running via [Docker Compose]. As time permits
-additional tooling will be added to support [Kubernetes], and deploying to
-[Amazon Elastic Container Service], and perhaps others.
-
-The example [docker-compose.yml] runs the [sandbox](./sandbox/README.md) version
-of Islandora, for the purposes of testing the images, and Islandora. When
-creating a ``docker-compose.yml`` for running a production Islandora 8 site. you
-will use your own image. Please see
-[Customizing the Drupal Installation](#customizing-the-drupal-installation) for
-instructions on how to do so.
-
-In addition to the images provided as described in the section
-[Docker Images](#docker-images). Several others are used by the
-[docker-compose.yml] file.
-
-### Watchtower
-
-The [watchtower](https://hub.docker.com/r/v2tec/watchtower/) container monitors
-the running Docker containers and watches for changes to the images that those
-containers were originally started from. If watchtower detects that an image has
-changed, it will automatically restart the container using the new image. This
-allows for automatic deployment, and overall faster development time.
-
-Note however Watchtower will not restart stopped container or containers that
-exited due to error. To ensure a container is always running, unless explicitly
-stopped, add ``restart: unless-stopped`` property to the container in the
-[docker-compose.yml] file. For example:
-
-```yaml
-database:
-    image: islandora/mariadb:latest
-    restart: unless-stopped
-```
-
-### Traefik
-
-The [traefik](https://containo.us/traefik/) container acts as a reverse proxy,
-and exposes some containers through port ``80`` on the localhost via the
-[loopback](https://www.tldp.org/LDP/nag/node66.html). This allows access to the
-following urls.
-
-- <http://activemq.localhost/admin>
-- <http://blazegraph.localhost/bigdata>
-- <http://islandora.localhost>
-- <http://fcrepo.localhost/fcrepo/reset>
-- <http://matomo.localhost>
-
-Note if you cannot map ``traefik`` to the hosts port 80, you will need to
-manually modify your ``/etc/hosts`` file and add entries for each of the urls
-above like so, assuming the IP of ``traefik`` container is ``x.x.x.x`` on its
-virtual network, and you can access that address from your machine.
-
-```properties
-x.x.x.x     activemq.localhost
-x.x.x.x     blazegraph.localhost
-x.x.x.x     islandora.localhost
-x.x.x.x     fcrepo.localhost
-x.x.x.x     matomo.localhost
-```
-
-Since Drupal passes its ``Base URL`` along to other services in AS2 as a means
-of allowing them to find their way back. As well as having services like Fedora
-exposed at the same URL they are accessed by the micro-services to end users. We
-need to allow containers within the network to be accessible via the same URL,
-though not by routing through ``traefik`` since it is and edge router.
-
-So alias like the following are defined:
-
-```yaml
-drupal:
-    image: islandora/sandbox:latest
-    # ...
-    networks:
-      default:
-        aliases:
-          - drupal.localhost
-```
-
-These are set on the ``default`` network as that is the internal network (no
-access to the outside) on which all containers reside.
-
-## ETCD
-
-The [etcd](https://github.com/etcd-io/etcd) container is a distributed reliable
-key-value store, which this project uses for configuration settings and secrets.
-Chosen in particular for it's existing integration with
-[Kubernetes](https://kubernetes.io/docs/concepts/overview/components/#etcd).
-
-Alternatively if removed from the [docker-compose.yml] file or explicitly not
-started the containers will fall back to pulling configuration from
-**environment variables**.
-
-A convenience script is provided that allows for users to put and get key/values
-from the store after it has been started. For example changing the log level of
-[houdini](./houdini/README.md) to ``DEBUG``.
-
-```bash
-./commands/etcdctl.sh put /houdini/log/level DEBUG
-```
-
-Or checking what the current log level is set to (*if not set to the default, in
-which case the key/value store is not used*):
-
-```bash
-./commands/etcdctl.sh get /houdini/log/level
-```
-
-## Customizing the Drupal Installation
-
-This needs to be thought about more in-depth, and needs fleshing out. Ideally we
-will provide the base image for building / deploying the Drupal image. End users
-will consume these containers and provide their ``composer.json`` and
-``composer.lock`` files, along with the name of the
-[Drupal Installation Profile] (either the one we will provide or one they create
-on their own). At the moment the [composer](./composer/README.md) project is
-provided as an early example of this. Where it user the
-``islandora/nginx as compose`` image to perform do the composer installation,
-and ``islandora/drupal`` as the container to run the installation created in the
-previous step.
-
-Additionally more documentation is needed to describe how developers can use the
-existing images for local development allowing them to create their own
-``composer.json`` and ``composer.lock`` files as well as any custom modules,
-themes, etc.
-
 ## Design Considerations
 
 All of the images build by this project are derived from the
@@ -387,7 +201,8 @@ The [base](./base/README.md) image includes two tools essential to the
 functioning of all the images.
 
 - [Confd](https://github.com/kelseyhightower/confd) - Configuration Management
-- [S6 Overlay](https://github.com/just-containers/s6-overlay) - Process Manager / Initialization system.
+- [S6 Overlay](https://github.com/just-containers/s6-overlay) - Process Manager
+  / Initialization system.
 
 ### Confd
 
@@ -498,7 +313,6 @@ are arranged in a hierarchy, that roughly follows below:
     │       └── fits
     ├── mariadb
     └── nginx
-        ├── composer
         ├── crayfish
         │   ├── gemini
         │   ├── homarus
@@ -508,7 +322,7 @@ are arranged in a hierarchy, that roughly follows below:
         │   └── recast
         ├── crayfits
         ├── drupal
-        │   └── sandbox
+        │   └── demo
         └── matomo
 ```
 
@@ -522,8 +336,8 @@ To make reasoning about what files go where each image follows the same
 filesystem layout for copying files into the image.
 
 A folder called ``rootfs`` maps directly onto the linux filesystem of the final
-image. So for example ``rootfs/opt/islandora/configs`` will be
-``/opt/islandora/configs`` in the generated image.
+image. So for example ``rootfs/etc/islandora/configs`` will be
+``/etc/islandora/configs`` in the generated image.
 
 ### Build System
 
@@ -551,14 +365,14 @@ wether it be ``etcd``, ``consul``, or ``environment variables``, containers can
 successfully start without any other container present.
 
 This does not completely remove dependencies between containers, for example,
-when the [sandbox](../docker/sandbox/README.md) starts it requires a running
+when the [demo](../docker/demo/README.md) starts it requires a running
 [fcrepo](../docker/fcrepo/README.md) to be able to ingest nodes created by
 ``islandora_default`` features. In these cases an initialization script can
 block until another container is available or a timeout has been reached. For
 example:
 
 ```bash
-local fcrepo_host="{{ getv "/fcrepo/host" "fcrepo.localhost" }}"
+local fcrepo_host="{{ getv "/fcrepo/host" "fcrepo.isle-dc.localhost" }}"
 local fcrepo_port="{{ getv "/fcrepo/host" "80" }}"
 local fcrepo_url=
 
@@ -594,30 +408,4 @@ failed to solve with frontend dockerfile.v0: failed to solve with frontend gatew
 [Overlay2](https://docs.docker.com/storage/storagedriver/overlayfs-driver/#configure-docker-with-the-overlay-or-overlay2-storage-driver)
 filesystem with Docker. If that doesn't work trying building [without BuildKit](#building-without-buildkit).
 
-## To Do
-
-- Blazegraph isn't working
-- Check if Cantaloupe is working
-- Add support for multiple backends to fedora (currently only file is being used)
-- Confirm all derivative generation is working and check if additional tools
-  need to be build or custom builds of say `poppler utils` or some other tool,
-  etc are needed (need to test against a variety of inputs for each mimetype as
-  there may be some edge case).
-- Change public/private key generation to not rely on shared volume use the configuration management
-- Do we need to support tls for etcd? Probably not since it shouldn't be exposed outside of the network. Though would be nice.
-- Change solr configuration to no rely on shared volume
-- Ideally no shared volumes as then container can be more easily moved between nodes in a cluster
-- Get working under ECS/EKS
-- Get working with Kubernetes with auto scaling for micro-services
-- Developer workflow documentation / examples
-
-[Amazon Elastic Container Service]: https://aws.amazon.com/ecs/
 [Buildkit]: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md
-[Docker Compose]: https://docs.docker.com/compose/
-[docker-compose.yml]: ./docker-compose.yml
-[Drupal Installation Profile]: https://www.drupal.org/docs/8/distributions/creating-distributions/how-to-write-a-drupal-8-installation-profile
-[drush]: https://drushcommands.com/
-[etcdctrl]: https://etcd.io/docs/v3.4.0/dev-guide/interacting_v3/
-[Kubernetes]: https://kubernetes.io/
-[mysql]: https://dev.mysql.com/doc/refman/8.0/en/mysql.html
-