Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

node details overrides + logging utils #415

Merged
merged 44 commits into from
Feb 23, 2024
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
9c67f68
node details overrides + logging utils
fmigneault Dec 15, 2023
4a5ed09
remove old config
fmigneault Dec 15, 2023
adda409
update changes
fmigneault Dec 15, 2023
6be39d8
update unittests with new birdhouse/scripts/read-configs.include.sh path
fmigneault Dec 15, 2023
805e5da
adjust cd exit 1 code
fmigneault Dec 15, 2023
816292b
replace all THIS_FILE with readlink|realpath cmd
fmigneault Dec 15, 2023
a41dd91
replace backticks by $(...) for path resolution operations
fmigneault Dec 19, 2023
d9fb457
fix canarie-api config conflict
fmigneault Dec 19, 2023
eff7dd3
update changes
fmigneault Jan 8, 2024
1d30103
add log function and logging level support
fmigneault Jan 8, 2024
6a75a10
add security defaults check for required variables that should be mod…
fmigneault Jan 11, 2024
8692201
add note about deprecated vars defaults
fmigneault Jan 12, 2024
c92e6a0
jupyterhub: ensure username variables are respected everywhere
mishaschwartz Jan 16, 2024
92c3fab
Merge branch 'master' into fix-node-details
fmigneault Jan 19, 2024
4fd9d01
ensure change is backwards compatible
mishaschwartz Jan 22, 2024
7390245
unittest
mishaschwartz Jan 22, 2024
7ea0f63
adjust sourced config file to standardize across scripts
fmigneault Jan 23, 2024
2d120a1
remove duplicate nested include of logging utils
fmigneault Jan 23, 2024
7314005
adjust pre/post compose script to include -x command display only on …
fmigneault Jan 23, 2024
17c5a94
add double quotes around default values
fmigneault Jan 23, 2024
ec77c07
Merge branch 'fix-node-details' of https://github.com/bird-house/bird…
fmigneault Jan 23, 2024
88e8ab3
Merge branch 'master' into fix-node-details
fmigneault Jan 23, 2024
58b2c4c
revert canarie config invalid template vars values
fmigneault Jan 23, 2024
1c8b615
Merge branch 'fix-node-details' into security-defaults
mishaschwartz Jan 23, 2024
ba2c863
Merge branch 'fix-node-details' of https://github.com/bird-house/bird…
fmigneault Jan 23, 2024
3cae728
replace missed logging call
fmigneault Jan 25, 2024
24950ba
Magpie: ensure that the `MAGPIE_ADMIN_USERNAME` variable is respected…
mishaschwartz Jan 25, 2024
adb54c7
add check_default_vars to read_basic_configs_only
fmigneault Jan 25, 2024
1e53f57
Merge branch 'fix-node-details' of https://github.com/bird-house/bird…
fmigneault Jan 25, 2024
22a161f
replace check_required_vars exit by return
fmigneault Jan 25, 2024
e17649b
add comments from PR review
fmigneault Jan 25, 2024
bcfe94c
move discover_compose_dir to resolve COMPOSE_DIR and reuse it to sour…
fmigneault Jan 26, 2024
f59a1e1
fix missing echo -> log conversions
fmigneault Jan 26, 2024
151c8a5
adjustments from PR review
fmigneault Jan 26, 2024
13392c8
move compose list resolution and 'info' mode handling before pre-dock…
fmigneault Jan 26, 2024
ccb97b5
Merge branch 'master' into fix-node-details
mishaschwartz Feb 16, 2024
193bdb1
Merge branch 'master' into fix-node-details
mishaschwartz Feb 16, 2024
85d2910
do not export __DEFAULT__ variables
fmigneault Feb 20, 2024
dbc79a3
reuse __DEFAULT__ values for env.local.example
fmigneault Feb 20, 2024
442d2f2
add COMPOSE_DIR check if provided explicitly
fmigneault Feb 20, 2024
9649cf3
revert read-configs.include.sh move + unify variables/escapes/quote-exec
fmigneault Feb 20, 2024
85a258a
fix read-configs.include.sh location in unittests
fmigneault Feb 21, 2024
ebf070f
adjust comments about __DEFAULT__ variables
fmigneault Feb 21, 2024
0dbd31d
Bump version: 2.0.6 → 2.1.0
fmigneault Feb 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,4 @@ venv/

## Testing
.pytest_cache/
*.log
35 changes: 33 additions & 2 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,44 @@
[Unreleased](https://github.com/bird-house/birdhouse-deploy/tree/master) (latest)
------------------------------------------------------------------------------------------------------------------

## Fixes
## Changes
- Compose script utilities:
* Add `BIRDHOUSE_COLOR` option and various logging/messaging definitions in `birdhouse/scripts/logging.include.sh`.
* Replace all explicit color "logging" related `echo` in scripts by a utility `log {LEVEL} {message}` function
that employs variables `LOG_DEBUG`, `LOG_INFO`, `LOG_WARN`, `LOG_ERROR` and `LOG_CRITICAL` as applicable per
respective messages to report logging messages in a standard approach.
Colors can be disabled with `BIRDHOUSE_COLOR=0` and logging level can be set with `BIRDHOUSE_LOG_LEVEL={LEVEL}`
where all levels above or equal to the configured one will be displayed (default logging level is `INFO`).
* Unify all `birdhouse/scripts` utilities to employ the same `COMPOSE_DIR` variable (auto-resolved or explicitly set)
in order to include or source any relevant dependencies they might have within the `birdhouse-deploy` repository.
* Add `info` option (ie: `pavics-compose.sh info`) that will stop processing just before `docker-compose` call.
This can be used to perform a "dry-run" of the command and validate that was is loaded is as expected, by inspecting
provided log messages.
* Replace older backtick (``` ` ```) executions by `$(...)` representation except for `eval` calls that require
them for backward compatibility of `sh` on some server instances.
* Modify the `sh -x` calls to scripts listed in `COMPONENT_PRE_COMPOSE_UP` and `COMPONENT_POST_COMPOSE_UP` to employ
the `-x` flag (showing commands) only when `BIRDHOUSE_LOG_LEVEL=DEBUG`.

- Defaults:
* Add multiple `SERVER_[...]` variables with defaults using previously hard coded values referring to PAVICS.
These variables use a special combination of `DELAYED_EVAL` and `OPTIONAL_VARS` definitions that can make use
of a variable formatted as `<ANY_NAME>='${__DEFAULT__<ANY_NAME>}'` that will print a warning messages indicating
that the default is employed, although *STRONGLY* recommended to be overridden. This allows a middle ground between
backward-compatible `env.local` while flagging potentially misused configurations.

## Fixes
- Canarie-API: updated references
* Use the new `SERVER_[...]` variables.
* Replace the LICENSE URL of the server node pointing
at [Ouranosinc/pavics-sdi](https://github.com/Ouranosinc/pavics-sdi) instead
of intended [bird-house/birdhouse-deploy](https://github.com/bird-house/birdhouse-deploy).
- Magpie: ensure that the `MAGPIE_ADMIN_USERNAME` variable is respected
* When determining the `JUPYTERHUB_ADMIN_USERS` variable
* Double check that it is being respected everywhere else
- env.local.example: fix `JUPYTERHUB_CONFIG_OVERRIDE` comment section

`JUPYTERHUB_CONFIG_OVERRIDE` was disconnected from its sample code.


[2.0.6](https://github.com/bird-house/birdhouse-deploy/tree/2.0.6) (2024-02-15)
------------------------------------------------------------------------------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion birdhouse/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ instructions below.
Manual instructions:

* Go to
``https://<PAVICS_FQDN>/magpie/ui/login`` and login with the ``admin`` user. The password should be in ``env.local``.
``https://<PAVICS_FQDN>/magpie/ui/login`` and login with the ``MAGPIE_ADMIN_USERNAME`` user. The password should be in ``env.local``.

* Then go to ``https://<PAVICS_FQDN>/magpie/ui/users/add``.

Expand Down
2 changes: 1 addition & 1 deletion birdhouse/components/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@ birdhouse-deploy software stack and the machine that it is running on. It is hig
make these routes available to anyone who does not have proper access permissions.

Add existing users to the ``monitoring`` group to allow them access to the various monitoring WebUI.
This way, we do not need to share the ``admin`` user account and do not have to add them to the
This way, we do not need to share the ``MAGPIE_ADMIN_USERNAME`` user account and do not have to add them to the
``administrators`` group, which would give them too much permissions.


Expand Down
20 changes: 10 additions & 10 deletions birdhouse/components/canarie-api/docker_configuration.py.template
Original file line number Diff line number Diff line change
Expand Up @@ -111,23 +111,23 @@ SERVICES = {
# Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
'version': '2.0.6',
'releaseTime': '2024-02-15T16:29:01Z',
'institution': 'Ouranos',
'researchSubject': 'Climatology',
'institution': '${SERVER_INSTITUTION}',
'researchSubject': '${SERVER_SUBJECT}',
'supportEmail': '${SUPPORT_EMAIL}',
'category': 'Resource/Cloud Management',
'tags': ['Climatology']
'tags': [tag.strip() for tag in "${SERVER_TAGS}".split(",") if tag.strip()],
},
'stats': {
'method': '.*',
'route': '(?!)' # this will be set by CANARIE_STATS_ROUTES (see below)
},
'redirect': {
'doc': 'https://pavics-sdi.readthedocs.io/en/latest/arch/backend.html',
'releasenotes': 'https://github.com/bird-house/birdhouse-deploy/blob/master/CHANGES.md',
'support': 'https://github.com/bird-house/birdhouse-deploy/issues',
'doc': '${SERVER_DOCUMENTATION_URL}',
'releasenotes': '${SERVER_RELEASE_NOTES_URL}',
'support': '${SERVER_SUPPORT_URL}',
'source': 'https://github.com/bird-house/birdhouse-deploy',
'tryme': 'https://${PAVICS_FQDN_PUBLIC}',
'licence': 'https://pavics-sdi.readthedocs.io/en/latest/license.html',
'licence': '${SERVER_LICENSE_URL}',
'provenance': 'https://pavics-sdi.readthedocs.io/en/latest/provenance/index.html'
},
'monitoring': {} # filled in after processing everything, see end of script
Expand All @@ -144,10 +144,10 @@ PLATFORMS = {
# Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
'version': '2.0.6',
'releaseTime': '2024-02-15T16:29:01Z',
'institution': 'Ouranos',
'researchSubject': 'Climatology',
'institution': '${SERVER_INSTITUTION}',
'researchSubject': '${SERVER_SUBJECT}',
'supportEmail': '${SUPPORT_EMAIL}',
'tags': ['Climatology', 'Cloud']
'tags': [tag.strip() for tag in "${SERVER_TAGS}".split(",") if tag.strip()],
},
'stats': {
'method': '.*',
Expand Down
17 changes: 12 additions & 5 deletions birdhouse/components/geoserver/pre-docker-compose-up
Original file line number Diff line number Diff line change
@@ -1,10 +1,17 @@
#!/bin/sh

THIS_FILE="`realpath "$0"`"
THIS_DIR="`dirname "$THIS_FILE"`"
COMPOSE_DIR="$THIS_DIR/../.."
THIS_FILE="$(readlink -f "$0" || realpath "$0")"
THIS_DIR="$(dirname "${THIS_FILE}")"
COMPOSE_DIR="${COMPOSE_DIR:-$(dirname "${THIS_DIR}/..")}"

if [ -f "${COMPOSE_DIR}/read-configs.include.sh" ]; then
. "${COMPOSE_DIR}/read-configs.include.sh"

# resolve GEOSERVER_DATA_DIR
read_configs
fi

if [ ! -f "${GEOSERVER_DATA_DIR}/global.xml" ]; then
echo "fix GeoServer data dir permission on first run only, when data dir do not exist yet."
FIRST_RUN_ONLY=1 "$COMPOSE_DIR"/deployment/fix-geoserver-data-dir-perm
log INFO "fix GeoServer data dir permission on first run only, when data dir do not exist yet."
FIRST_RUN_ONLY=1 "${COMPOSE_DIR}/deployment/fix-geoserver-data-dir-perm"
fi
4 changes: 4 additions & 0 deletions birdhouse/components/jupyterhub/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,13 @@ export JUPYTERHUB_CRYPT_KEY=
# JUPYTERHUB_CRYPT_KEY is set.
export JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE=60

# Usernames that should be given admin access in jupyterhub
export JUPYTERHUB_ADMIN_USERS='{\"${MAGPIE_ADMIN_USERNAME}\"}' # python set syntax

export DELAYED_EVAL="
$DELAYED_EVAL
JUPYTERHUB_USER_DATA_DIR
JUPYTERHUB_ADMIN_USERS
"

# add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
Expand Down
5 changes: 2 additions & 3 deletions birdhouse/components/proxy/pre-docker-compose-up
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,11 @@
# Create JSON files containing the version information, available services, and
# enabled components. These files will be served by the nginx proxy as static files.

THIS_FILE="$(realpath "$0")"
THIS_DIR="$(dirname "$THIS_FILE")"
THIS_FILE="$(readlink -f "$0" || realpath "$0")"
THIS_DIR="$(dirname "${THIS_FILE}")"

mkdir -p "${THIS_DIR}/static"

echo "${BIRDHOUSE_VERSION_JSON}" > "${THIS_DIR}/static/version.json"
echo "${BIRDHOUSE_DEPLOY_SERVICES_JSON}" > "${THIS_DIR}/static/services.json"
echo "${BIRDHOUSE_DEPLOY_COMPONENTS_JSON}" > "${THIS_DIR}/static/components.json"

80 changes: 77 additions & 3 deletions birdhouse/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@
# must use single quotes to avoid early expansion before overrides in env.local
# are applied and must be added to the list of DELAYED_EVAL.

# Any default value that should be marked for security concern or recommended modificiation should
# use the '${__DEFAULT__{var}}' naming format. These can then be referenced in 'env.local.example' to
# avoid literal value duplication, and ensure they remain in sync. Also, those '${__DEFAULT__{var}}'
# definitions should *NOT* be exported to avoid unnecessary polution of the environment variables.
# Variables with format '${__DEFAULT__{var}}' will be flagged accordingly to their required/optional status
# (see also: 'check_default_vars' in 'birdhouse/read-configs.include.sh').

export BASH_IMAGE="bash:5.1.4"

# Root directory under which all data persistence should be nested under
Expand All @@ -29,19 +36,86 @@ export PAVICS_FQDN_PUBLIC='${PAVICS_FQDN}'
export DELAYED_EVAL="
$DELAYED_EVAL
PAVICS_FQDN_PUBLIC
DOC_URL
SUPPORT_EMAIL
SSL_CERTIFICATE
DATA_PERSIST_SHARED_ROOT
WPS_OUTPUTS_DIR
SERVER_NAME
SERVER_DESCRIPTION
SERVER_INSTITUTION
SERVER_SUBJECT
SERVER_TAGS
SERVER_DOCUMENTATION_URL
SERVER_RELEASE_NOTES_URL
SERVER_SUPPORT_URL
SERVER_LICENSE_URL
"


export SERVER_NAME=PAVICS
export SERVER_DESCRIPTION="
# Server Identification Details
# Following definitions should definitenly be updated.
# Previous defaults are defined for backward-compatibility.
# If not overridden explicitly by their non '__' prefixed variant,
# a WARN message will be displayed by pavics-compose.
__DEFAULT__SERVER_NAME="PAVICS"
__DEFAULT__SERVER_DESCRIPTION="
The PAVICS (Power Analytics for Visualization of Climate Science) platform is a collection of
climate analysis services served through Open Geospatial Consortium (OGC) protocols.
These services include data access, processing and visualization. Both data and algorithms
can be accessed either programmatically, through OGC-compliant clients such as QGIS or ArcGIS,
or a custom web interface.
"
__DEFAULT__SERVER_INSTITUTION="Ouranos"
__DEFAULT__SERVER_SUBJECT="Climatology"
# below can be a CSV list of tags
__DEFAULT__SERVER_TAGS="Climatology"
__DEFAULT__SERVER_DOCUMENTATION_URL="https://pavics-sdi.readthedocs.io/en/latest/arch/backend.html"
__DEFAULT__SERVER_RELEASE_NOTES_URL="https://github.com/bird-house/birdhouse-deploy/blob/master/CHANGES.md"
__DEFAULT__SERVER_SUPPORT_URL="https://github.com/bird-house/birdhouse-deploy/issues"
# NOTE:
# This value does not use the previously hard coded default.
# Previous default pointed at the wrong repository with a mismatching LICENSE file.
__DEFAULT__SERVER_LICENSE_URL="https://github.com/bird-house/birdhouse-deploy/blob/master/LICENSE"
__DEFAULT__SUPPORT_EMAIL="[email protected]"
__DEFAULT__DOC_URL="https://www.example.com/"
__DEFAULT__PAVICS_FQDN="hostname.domainname"
__DEFAULT__SSL_CERTIFICATE="/path/to/ssl/cert.pem"

# apply overrides or fallback above defaults with delayed evaluation
# exceptions for 'SUPPORT_EMAIL' and 'DOC_URL' using the old name for backward compatibility.
export SUPPORT_EMAIL='${__DEFAULT__SUPPORT_EMAIL}'
export DOC_URL='${__DEFAULT__DOC_URL}'
export SSL_CERTIFICATE='${__DEFAULT__SSL_CERTIFICATE}'
export SERVER_NAME='${__DEFAULT__SERVER_NAME}'
export SERVER_DESCRIPTION='${__DEFAULT__SERVER_DESCRIPTION}'
export SERVER_INSTITUTION='${__DEFAULT__SERVER_INSTITUTION}'
export SERVER_SUBJECT='${__DEFAULT__SERVER_SUBJECT}'
export SERVER_TAGS='${__DEFAULT__SERVER_TAGS}'
export SERVER_DOCUMENTATION_URL='${__DEFAULT__SERVER_DOCUMENTATION_URL}'
export SERVER_RELEASE_NOTES_URL='${__DEFAULT__SERVER_RELEASE_NOTES_URL}'
export SERVER_SUPPORT_URL='${__DEFAULT__SERVER_SUPPORT_URL}'
export SERVER_LICENSE_URL='${__DEFAULT__SERVER_LICENSE_URL}'

# Defaults for required variables recommended for override for security reasons.
# Those will not be set explicitly as defaults to ensure they are overridden explicitly by the instance.
# These values would be detected only if the instance was configured using a copy of 'env.local.example'.
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
__DEFAULT__MAGPIE_SECRET="itzaseekrit"
__DEFAULT__MAGPIE_ADMIN_USERNAME="admin"
__DEFAULT__MAGPIE_ADMIN_PASSWORD="qwertyqwerty!"
__DEFAULT__POSTGRES_PAVICS_USERNAME="postgres-pavics"
__DEFAULT__POSTGRES_PAVICS_PASSWORD="postgres-qwerty"
__DEFAULT__POSTGRES_MAGPIE_USERNAME="postgres-magpie"
__DEFAULT__POSTGRES_MAGPIE_PASSWORD="postgres-qwerty"
__DEFAULT__GEOSERVER_ADMIN_USER="admingeo"
__DEFAULT__GEOSERVER_ADMIN_PASSWORD="geoserverpass"
#############################################################################
# Deprecated vars (for components in the ./deprecated-components directory)
#############################################################################
__DEFAULT__TOMCAT_NCWMS_PASSWORD="ncwmspass"
__DEFAULT__CATALOG_USERNAME="admin-catalog"
__DEFAULT__CATALOG_PASSWORD="qwerty"
__DEFAULT__PHOENIX_PASSWORD="phoenix_pass"
__DEFAULT__PHOENIX_PASSWORD_HASH="sha256:123456789012:1234567890123456789012345678901234567890123456789012345678901234"

export DEFAULT_CONF_DIRS='
./components/proxy
Expand Down
4 changes: 2 additions & 2 deletions birdhouse/deployment/certbotwrapper
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ certbotwrapper START_TIME=$START_TIME"

set -x

THIS_FILE="`realpath "$0"`"
THIS_DIR="`dirname "$THIS_FILE"`"
THIS_FILE="$(readlink -f "$0" || realpath "$0")"
THIS_DIR="$(dirname "${THIS_FILE}")"
SAVED_PWD="`pwd`"

. "$THIS_DIR/../read-configs.include.sh"
Expand Down
Loading
Loading