Skip to content

Commit

Permalink
LIBDRUM-889. Make DRUM local dev setup more consistent with MD-SOAR
Browse files Browse the repository at this point in the history
Made the local development setup more consistent with MD-SOAR:

* Renamed "dspace/docs/Drum7DockerDevelopmentEnvironment.md" to
  "dspace/docs/DockerDevelopmentEnvironment.md"
* Renamed the "postgres-init/pg_restore_drum.sh" script to
  "postgres-init/pg_restore.sh" and modify it to use the "POSTGRES_DB"
  environment variable, instead of hard-coding the database
* Updated the documentation to reflect the renaming of the
  "pg_restore.sh" script
* Added information to the "postgres-init/README.md" on verifying that
  a database dump from Kubernetes is a complete snapshot

https://umd-dit.atlassian.net/browse/LIBDRUM-889
  • Loading branch information
dsteelma-umd committed Nov 1, 2024
1 parent a77cfc9 commit dde576e
Show file tree
Hide file tree
Showing 7 changed files with 57 additions and 27 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dspace/src/main/docker/nginx/certs/*
# Ignore database dump directory to be used with docker-compose for initializing
postgres-init/*
!postgres-init/README.md
!postgres-init/pg_restore_drum.sh
!postgres-init/pg_restore.sh
## End UMD Customization

## Ignore the MVN compiled output directories from version tracking
Expand Down
2 changes: 1 addition & 1 deletion README-DRUM.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The original DSpace documentation is in the "README.md" file.
## Development Environment

Instructions for building and running drum locally can be found in
[dspace/docs/Drum7DockerDevelopmentEnvironment.md](/dspace/docs/Drum7DockerDevelopmentEnvironment.md)
[dspace/docs/DockerDevelopmentEnvironment.md](/dspace/docs/DockerDevelopmentEnvironment.md)

## Building Images for K8s Deployment

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# DRUM 7 Docker Development Environment
# Docker Development Environment

This document contains instructions for building a local development instance
of a DSpace 7-based DRUM using Docker.
Expand Down Expand Up @@ -51,8 +51,8 @@ and add the following lines:

## Development Setup

This repository uses the "GitHub Flow" branching model, with "drum-main" as the
main branch for DRUM development.
This repository uses the "GitHub Flow" branching model, with "drum-main" as
the main branch for DRUM development.

1) Clone the Git repository and switch to the directory:

Expand All @@ -79,23 +79,24 @@ main branch for DRUM development.

4) Edit the local configuration file:

```zsh
$ vi dspace/config/local.cfg
```
```zsh
$ vi dspace/config/local.cfg
```

and enter values for the following properties:
and enter values for the following properties:

* drum.ldap.bind.auth
* drum.ldap.bind.password
* drum.ldap.bind.auth
* drum.ldap.bind.password

The appropriate values can be found in LastPass.
The appropriate values can be found in LastPass.

**Note:** The "drum.ldap.bind.auth" value typically contains commas (for
example "uid=foo,cn=bar,ou=baz,dc=quuz,dc=zot"), which must be escaped. So
the actual value added to the file would be similar to
`uid=foo\,cn=bar\,ou=baz\,dc=quuz\,dc=zot`.
**Note:** The "drum.ldap.bind.auth" value typically contains commas (for
example "uid=foo,cn=bar,ou=baz,dc=quuz,dc=zot"), which must be escaped. So
the actual value added to the file would be similar to
`uid=foo\,cn=bar\,ou=baz\,dc=quuz\,dc=zot`.

5) Follow the instructions at [dspace/docs/DrumDBRestore.md](DrumDBRestore.md)
5) Follow the instructions at
[dspace/docs/DrumDBRestore.md](DrumDBRestore.md)
to populate the Postgres database with a DSpace 7 database dump from
Kubernetes.

Expand All @@ -116,7 +117,6 @@ main branch for DRUM development.
```zsh
# Build the dspace image
$ docker compose -f docker-compose.yml build
```

8) Start all the containers
Expand Down Expand Up @@ -155,6 +155,9 @@ $ docker compose -p d7 up -d dspacedb dspacesolr nginx
$ docker compose -p d7 up dspace
```

Once the REST API starts, it should be accessible at
<https://api.drum-local.lib.umd.edu/server>

## Visual Studio Code IDE Setup

The following is the suggested setup for Visual Studio Code for DSpace
Expand Down
2 changes: 1 addition & 1 deletion dspace/docs/DrumConfigurationCustomization.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ In general, changes *should not* be made to this file.

DRUM customization should be added to the "dspace/config/local.cfg.EXAMPLE"
file, which, when deployed, should be copied to "dspace/config/local.cfg" (see
the [DRUM 7 Docker Development Environment](Drum7DockerDevelopmentEnvironment.md).
the [Docker Development Environment](DockerDevelopmentEnvironment.md).

## XML File Changes

Expand Down
4 changes: 2 additions & 2 deletions dspace/docs/DrumDBRestore.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ with DRUM running DSpace 7.
```

**Note:** The output file MUST use a ".dump" extension, in order for the
"pg_restore_drum.sh" script to process it, see
"pg_restore.sh" script to process it, see
"[postgres-init/README.md](../../postgres-init/README.md)"

3) (Optional) This step can be skipped, if you are following the instructions in
[dspace/docs/Drum7DockerDevelopmentEnvironment.md](Drum7DockerDevelopmentEnvironment.md).
[dspace/docs/DockerDevelopmentEnvironment.md](DockerDevelopmentEnvironment.md).

Start the "dspacedb" container and wait for the restore to complete.

Expand Down
37 changes: 32 additions & 5 deletions postgres-init/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Database initialization for docker-compose

Place your database dump file in this directory, and the "dspacedb" container
will use it to initialize the database, via the "pg_restore_drum.sh" script.
will use it to initialize the database, via the "pg_restore.sh" script.

## pg_restore_drum.sh and CVE-2023-2454
## pg_restore.sh and CVE-2023-2454

Prior to DSpace 7.6.2, it was possible to use "pg_dump" to create a "plain" SQL
file (with a ".sql" extension), and the "postgres" Docker image
Expand All @@ -16,12 +16,12 @@ process running as the "postgres" user, and the use of the "pgcrypto"
extension). Note that plain SQL dumps *will* work when run from the Docker
container's command-line.

The workaround was to create the "postgres-init/pg_restore_drum.sh", script
The workaround was to create the "postgres-init/pg_restore.sh", script
which uses "pg_restore" to populate the database with a Postgres "custom" dump
file. This is used in preference to a plain SQL file, to keep the automatic
Docker initialization from interfering with the process.

The "postgres-init/pg_restore_drum.sh" will run automatically when the Docker
The "postgres-init/pg_restore.sh" will run automatically when the Docker
container starts, *if* a database doesn't already exist.

## Creating the Postgres dump
Expand All @@ -33,7 +33,34 @@ $ kubectl exec drum-db-0 -- pg_dump -Fc -C -O -U drum -d drum > postgres-init/dr
```

**Note:** The output file MUST use a ".dump" extension, in order for the
"pg_restore_drum.sh" script to process it.
"pg_restore.sh" script to process it.

## (Optional) Verifying the Postgres dump

Use of the custom dump format is slightly risky because the Kubernetes
connection might have terminated before completing the dump.

Assuming that Postgres is installed on the local workstation, one way to
verify that the dump is complete, is to convert the database dump into a
plain SQL file (such as "verify-db.sql") by running:

```zsh
$ pg_restore -f verify-db.sql postgres-init/drum-db.dump
```

and then running the "tail" command against the plain SQL file:

```zsh
$ tail verify-db.sql
```

If the dump is complete, the last lines will be:

```text
--
-- PostgreSQL database dump complete
--
```

## Delete local database data

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd $SCRIPT_DIR

# Run pg_restore against any file in the directory with a ".dump" extension
find . -name '*.dump' -exec pg_restore --username="$POSTGRES_USER" --no-owner --dbname=drum --verbose {} \;
find . -name '*.dump' -exec pg_restore --username="$POSTGRES_USER" --no-owner --dbname="$POSTGRES_DB" --verbose {} \;

0 comments on commit dde576e

Please sign in to comment.