Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add node services URI and fields #445

Merged
merged 12 commits into from
Apr 18, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,19 @@
[Unreleased](https://github.com/bird-house/birdhouse-deploy/tree/master) (latest)
------------------------------------------------------------------------------------------------------------------

[//]: # (list changes here, using '-' for each new entry, remove this when items are added)
## Changes

- Node Services: Add definitions and variables for every service represented by
the [DACCS-Climate/Marble-node-registry](https://github.com/DACCS-Climate/Marble-node-registry).

- Add `version` field using the corresponding `<SERVICE>_VERSION` variables.
- Add `types` field restricted by specific values instead of previous `keywords` expected to be extendable.
- Add `<SERVICE>_IMAGE_URI` variables to provide `rel: service-meta` link for every service.
- Update all `$schema` references of service node registry
to [1.2.0](https://github.com/DACCS-Climate/Marble-node-registry/releases/tag/1.2.0) instead of `main`.
During unit tests, specific `$schema` reference in the respective service configuration will be used for validation.

See [bird-house/birdhouse-deploy#441](https://github.com/bird-house/birdhouse-deploy/issues/441) for more details.

[2.1.3](https://github.com/bird-house/birdhouse-deploy/tree/2.1.3) (2024-04-09)
------------------------------------------------------------------------------------------------------------------
Expand Down
13 changes: 12 additions & 1 deletion birdhouse/components/finch/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
# are applied and must be added to the list of DELAYED_EVAL.

export FINCH_VERSION=0.9.2
export FINCH_IMAGE="birdhouse/finch:version-${FINCH_VERSION}"
export FINCH_DOCKER=birdhouse/finch
export FINCH_IMAGE='${FINCH_DOCKER}:version-${FINCH_VERSION}'
export FINCH_IMAGE_URI='registry.hub.docker.com/${FINCH_IMAGE}'

# This tells the postgres service to create a database on startup
export POSTGRES_DATABASES_TO_CREATE="
Expand All @@ -25,7 +27,16 @@ COMPONENT_DEPENDENCIES="
./components/postgres
"

export DELAYED_EVAL="
$DELAYED_EVAL
FINCH_IMAGE
FINCH_IMAGE_URI
"

OPTIONAL_VARS="
$OPTIONAL_VARS
\$FINCH_VERSION
\$FINCH_DOCKER
\$FINCH_IMAGE
\$FINCH_IMAGE_URI
"
11 changes: 10 additions & 1 deletion birdhouse/components/finch/service-config.json.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/1.2.0/node_registry.schema.json#service",
"name": "finch",
"version": "${FINCH_VERSION}",
"types": [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hasn't been implemented yet in the schema. Adding this won't cause schema validation to fail but do we plan on updating the schema before or after this PR is merged?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the schema gets updated to include it beforehand, then we can bump its version right away. Otherwise, it can be done later. As mentioned, it doesn't cause an issue regardless at the moment since it is not validated.

"wps"
],
"keywords": [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we leave these keywords as is if we're moving this information to "types". What if we borrowed some of the tags and categories from canarie-api instead. So for finch we could do something like:

"keywords": ["Climatology", "Cloud", "Data Slicer", "Climate Indices", ...]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I would like to do this, but until there is a clear-cut version I can refer to, adding these to the keywords will fail schema validation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Try now with version 1.2.0?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to do this in this PR or in a different one?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, I'd prefer to work on this only once.

"service-wps"
],
Expand All @@ -20,6 +24,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/finch?service=WPS&request=GetCapabilities"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${FINCH_IMAGE_URI}"
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just my 2 cents, since the URI, do not start with http, should we name the new field "uri" instead of "href"? "href" would suggest starting with http usually. Ignore if it has to be "href" for compat or other reasons.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"href" is required by the schema so we can't change the name

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A non-http scheme value is a valid URI. It uniquely identifies the resource.
I would prefer to have a docker:// scheme or similar to be more explicit if permitted, but docker does not seem to like it. However, the values provided in the defaults can be used directly (e.g. docker pull 'registry.hub.docker.com/pavics/weaver:5.0.0' works, but http://registry.hub.docker.com/pavics/weaver:5.0.0 doesn't), so I find them more useful although maybe counter-intuitive.

Maybe one way to avoid confusion would be to use index.docker.io/pavics/weaver:5.0.0 instead, since it does not redirect to https://hub.docker.com/ like registry.hub.docker.com does? This alternate location still woks with docker pull 'index.docker.io/pavics/weaver:5.0.0', and it looks slightly more like the type definition. The idea of this link is really to have the exact value that can be plugged in the docker-compose service image field.

}
]
}
6 changes: 5 additions & 1 deletion birdhouse/components/geoserver/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
export GEOSERVER_DOCKER=pavics/geoserver
export GEOSERVER_VERSION=2.22.2
export GEOSERVER_TAGGED=2.22.2-kartoza-build20230226-r7-allow-change-context-root-and-fix-missing-stable-plugins-and-avoid-chown-datadir
export GEOSERVER_IMAGE="${GEOSERVER_DOCKER}:${GEOSERVER_TAGGED}"
export GEOSERVER_IMAGE='${GEOSERVER_DOCKER}:${GEOSERVER_TAGGED}'
export GEOSERVER_IMAGE_URI='registry.hub.docker.com/${GEOSERVER_IMAGE}'

export GEOSERVER_ADMIN_USER="admin"

Expand All @@ -34,6 +35,8 @@ export DELAYED_EVAL="
$DELAYED_EVAL
GEOSERVER_DATA_DIR
GEOSERVER_SKIP_AUTH_PROXY_INCLUDE
GEOSERVER_IMAGE
GEOSERVER_IMAGE_URI
"

# add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
Expand All @@ -49,5 +52,6 @@ OPTIONAL_VARS="
\$GEOSERVER_VERSION
\$GEOSERVER_TAGGED
\$GEOSERVER_IMAGE
\$GEOSERVER_IMAGE_URI
\$GEOSERVER_SKIP_AUTH_PROXY_INCLUDE
"
14 changes: 13 additions & 1 deletion birdhouse/components/geoserver/service-config.json.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/1.2.0/node_registry.schema.json#service",
"name": "geoserver",
"version": "${GEOSERVER_VERSION}",
"types": [
"data",
"wms",
"wfs",
"wps"
],
"keywords": [
"data",
"service-wms",
Expand All @@ -18,6 +25,11 @@
"rel": "service-doc",
"type": "text/html",
"href": "https://docs.geoserver.org/"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${GEOSERVER_IMAGE_URI}"
}
]
}
19 changes: 18 additions & 1 deletion birdhouse/components/hummingbird/default.env
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
export HUMMINGBIRD_VERSION=0.5_dev
# note: minor adjustments for 'version' field of service
# - version must include patch (https://semver.org/#spec-item-2)
# - pattern does not allow '_' (https://semver.org/#spec-item-9)
export HUMMINGBIRD_VERSION=0.5.0-dev
export HUMMINGBIRD_TAGGED=0.5_dev
export HUMMINGBIRD_DOCKER=pavics/hummingbird
export HUMMINGBIRD_IMAGE='${HUMMINGBIRD_DOCKER}:${HUMMINGBIRD_TAGGED}'
export HUMMINGBIRD_IMAGE_URI='registry.hub.docker.com/${HUMMINGBIRD_IMAGE}'

# This tells the postgres service to create a database on startup
export POSTGRES_DATABASES_TO_CREATE="
Expand All @@ -19,7 +26,17 @@ COMPONENT_DEPENDENCIES="
./components/data-volume
"

export DELAYED_EVAL="
$DELAYED_EVAL
HUMMINGBIRD_IMAGE
HUMMINGBIRD_IMAGE_URI
"

OPTIONAL_VARS="
$OPTIONAL_VARS
\$HUMMINGBIRD_VERSION
\$HUMMINGBIRD_DOCKER
\$HUMMINGBIRD_TAGGED
\$HUMMINGBIRD_IMAGE
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
\$HUMMINGBIRD_IMAGE_VERSION
"
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ x-logging:

services:
hummingbird:
image: pavics/hummingbird:${HUMMINGBIRD_VERSION}
image: ${HUMMINGBIRD_IMAGE}
container_name: hummingbird
environment:
HOSTNAME: $HOSTNAME
Expand Down
11 changes: 10 additions & 1 deletion birdhouse/components/hummingbird/service-config.json.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/1.2.0/node_registry.schema.json#service",
"name": "hummingbird",
"version": "${HUMMINGBIRD_VERSION}",
"types": [
"wps"
],
"keywords": [
"service-wps"
],
Expand All @@ -20,6 +24,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/hummingbird?service=WPS&request=GetCapabilities"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${HUMMINGBIRD_IMAGE_URI}"
}
]
}
6 changes: 6 additions & 0 deletions birdhouse/components/jupyterhub/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

export JUPYTERHUB_DOCKER=pavics/jupyterhub
export JUPYTERHUB_VERSION=4.1.4-20240408
export JUPYTERHUB_IMAGE='${JUPYTERHUB_DOCKER}:${JUPYTERHUB_VERSION}'
export JUPYTERHUB_IMAGE_URI='registry.hub.docker.com/${JUPYTERHUB_IMAGE}'

# Jupyter single-user server images, can be overriden in env.local to have a space separated list of multiple images
export DOCKER_NOTEBOOK_IMAGES="pavics/workflow-tests:py39-230601-1-update240116"
Expand Down Expand Up @@ -75,6 +77,8 @@ export DELAYED_EVAL="
$DELAYED_EVAL
JUPYTERHUB_USER_DATA_DIR
JUPYTERHUB_ADMIN_USERS
JUPYTERHUB_IMAGE
JUPYTERHUB_IMAGE_URI
"

# add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
Expand All @@ -93,6 +97,8 @@ OPTIONAL_VARS="
\$JUPYTERHUB_CONFIG_OVERRIDE
\$JUPYTERHUB_DOCKER
\$JUPYTERHUB_VERSION
\$JUPYTERHUB_IMAGE
\$JUPYTERHUB_IMAGE_URI
\$JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL
\$JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE
\$JUPYTER_IDLE_SERVER_CULL_TIMEOUT
Expand Down
2 changes: 1 addition & 1 deletion birdhouse/components/jupyterhub/docker-compose-extra.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ x-logging:

services:
jupyterhub:
image: ${JUPYTERHUB_DOCKER}:${JUPYTERHUB_VERSION}
image: ${JUPYTERHUB_IMAGE}
container_name: jupyterhub
hostname: jupyterhub
environment:
Expand Down
11 changes: 10 additions & 1 deletion birdhouse/components/jupyterhub/service-config.json.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/1.2.0/node_registry.schema.json#service",
"name": "jupyterhub",
"version": "${JUPYTERHUB_VERSION}",
"types": [
"jupyterhub"
],
"keywords": [
"jupyterhub"
],
Expand All @@ -15,6 +19,11 @@
"rel": "service-doc",
"type": "text/html",
"href": "https://jupyter.org/hub"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${JUPYTERHUB_IMAGE_URI}"
}
]
}
12 changes: 12 additions & 0 deletions birdhouse/components/raven/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
export RAVEN_GEO_URL="https://pavics.ouranos.ca/geoserver/"

export RAVEN_VERSION="0.18.1"
export RAVEN_DOCKER=pavics/raven
export RAVEN_IMAGE='${RAVEN_DOCKER}:${RAVEN_VERSION}'
export RAVEN_IMAGE_URI='registry.hub.docker.com/${RAVEN_IMAGE}'

# This tells the postgres service to create a database on startup
export POSTGRES_DATABASES_TO_CREATE="
Expand All @@ -27,7 +30,16 @@ COMPONENT_DEPENDENCIES="
./components/wps_outputs-volume
"

export DELAYED_EVAL="
$DELAYED_EVAL
RAVEN_IMAGE
RAVEN_IMAGE_URI
"

OPTIONAL_VARS="
$OPTIONAL_VARS
\$RAVEN_VERSION
\$RAVEN_DOCKER
\$RAVEN_IMAGE
\$RAVEN_IMAGE_VERSION
"
2 changes: 1 addition & 1 deletion birdhouse/components/raven/docker-compose-extra.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ x-logging:

services:
raven:
image: pavics/raven:${RAVEN_VERSION}
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
image: ${RAVEN_IMAGE}
container_name: raven
environment:
PYWPS_CFG: /wps.cfg
Expand Down
11 changes: 10 additions & 1 deletion birdhouse/components/raven/service-config.json.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/1.2.0/node_registry.schema.json#service",
"name": "raven",
"version": "${RAVEN_VERSION}",
"types": [
"wps"
],
"keywords": [
"service-wps"
],
Expand All @@ -20,6 +24,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/raven?service=WPS&request=GetCapabilities"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${RAVEN_IMAGE_URI}"
}
]
}
32 changes: 32 additions & 0 deletions birdhouse/components/stac/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,20 @@ export STAC_POSTGRES_PASSWORD=${POSTGRES_PAVICS_PASSWORD}
export STAC_PGUSER=${POSTGRES_PAVICS_USERNAME}
export STAC_PGPASSWORD=${POSTGRES_PAVICS_PASSWORD}

# 'main' branch points at https://github.com/stac-utils/stac-fastapi/commit/d53e792
# (see: https://github.com/crim-ca/stac-app/blob/40cad1aa7a094d58fca2d3184182761e248f781d/Dockerfile#L15-L20)
# which corresponds to 2.4.3 release
export STAC_VERSION=2.4.3-crim-main
export STAC_IMAGE='ghcr.io/crim-ca/stac-app:main'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why not push the image with the actual tag 2.4.3-crim instead of the generic tag main?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #371 and #346

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I could temporarily push a tag to make the reference more "semver"-like, but like @mishaschwartz highlighted, I would much rather have some work done on using the official image. The one tagged is already 9 months old. I don't want this patch tag to be considered a maintained revision.

export STAC_IMAGE_URI='${STAC_IMAGE}'

# 'docker_image_push' branch points at https://github.com/crim-ca/stac-browser/tree/docker_image_push
# which is some commits ahead of 'v3.0.0-beta.5' (and multiple behind latest official releases)
# version name is slightly tweaked to fulfill schema while leaving an obvious trace
export STAC_BROWSER_VERSION=3.0.0-beta.5-crim-docker-image-push
export STAC_BROWSER_IMAGE='ghcr.io/crim-ca/stac-browser:docker_image_push'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, 3.0.0-beta.5-crim instead of docker_image_push. It is allowed to tag while on a branch.

export STAC_BROWSER_IMAGE_URI='${STAC_BROWSER_IMAGE}'

# add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
# single quotes are important in below list to keep variable names intact until 'pavics-compose' parses them
EXTRA_VARS='
Expand All @@ -14,3 +28,21 @@ EXTRA_VARS='
# extend the original 'VARS' from 'birdhouse/pavics-compose.sh' to employ them for template substitution
# adding them to 'VARS', they will also be validated in case of override of 'default.env' using 'env.local'
VARS="$VARS $EXTRA_VARS"

export DELAYED_EVAL="
$DELAYED_EVAL
STAC_IMAGE
STAC_IMAGE_URI
STAC_BROWSER_IMAGE
STAC_BROWSER_IMAGE_URI
"

OPTIONAL_VARS="
$OPTIONAL_VARS
\$STAC_VERSION
\$STAC_IMAGE
\$STAC_IMAGE_URI
\$STAC_BROWSER_VERSION
\$STAC_BROWSER_IMAGE
\$STAC_BROWSER_IMAGE_URI
"
4 changes: 2 additions & 2 deletions birdhouse/components/stac/docker-compose-extra.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ x-logging:
services:
stac:
container_name: stac
image: ghcr.io/crim-ca/stac-app:main
image: ${STAC_IMAGE}
depends_on:
- stac-db
environment:
Expand All @@ -28,7 +28,7 @@ services:

stac-browser:
container_name: stac-browser
image: ghcr.io/crim-ca/stac-browser:docker_image_push
image: ${STAC_BROWSER_IMAGE}
environment:
- CATALOG_URL=https://${PAVICS_FQDN_PUBLIC}/stac/
- ROOT_PATH=/stac-browser/
Expand Down
Loading
Loading