Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add node services URI and fields #445

Merged
merged 12 commits into from
Apr 18, 2024
Merged

add node services URI and fields #445

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
41d6be8
add node services URI with 'rel:service-meta', 'version' field and 't…
fmigneault Apr 5, 2024
2310410
add missing OPTIONAL_VARGS + apply service config for flyingpigeon
fmigneault Apr 8, 2024
b6e3b5b
Merge branch 'master' into add-service-versions
mishaschwartz Apr 9, 2024
7522c40
add service VERSION and VERSION_URI variables to DELAYED_EVAL
fmigneault Apr 9, 2024
40126f7
update all service node schemas to 1.2.0
fmigneault Apr 9, 2024
f8b8357
Merge branch 'add-service-versions' of https://github.com/bird-house/…
fmigneault Apr 9, 2024
07f6b97
fix DELAYED_EVAL variable quotes + adjust tests to explicitly validat…
fmigneault Apr 10, 2024
485580a
cleanup test requirements + patch missing quotes/DELAYED_EVAL inconsi…
fmigneault Apr 10, 2024
9b0753f
use service-config lists everywhere
fmigneault Apr 15, 2024
546f9ac
fix optional-components/testthredds needing DELAYED_EVAL for THREDDS_…
fmigneault Apr 15, 2024
3b18f57
adjust canarie versions/tag mixup
fmigneault Apr 15, 2024
5249a6b
Bump version: 2.1.3 → 2.2.0
fmigneault Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,17 @@
[Unreleased](https://github.com/bird-house/birdhouse-deploy/tree/master) (latest)
------------------------------------------------------------------------------------------------------------------

## Changes

- Node Services: Add definitions and variables for every service represented by
the [DACCS-Climate/Marble-node-registry](https://github.com/DACCS-Climate/Marble-node-registry).

- Add `version` field using the corresponding `<SERVICE>_VERSION` variables.
- Add `types` field restricted by specific values instead of previous `keywords` expected to be extendable.
- Add `<SERVICE>_IMAGE_URI` variables to provide `rel: service-meta` link for every service.

See [bird-house/birdhouse-deploy#441](https://github.com/bird-house/birdhouse-deploy/issues/441) for more details.

## Fixes
- GeoServer: fix invalid media-type specified for the service's endpoint in `service-config.json.template`

Expand Down
3 changes: 3 additions & 0 deletions birdhouse/components/finch/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

export FINCH_VERSION=0.9.2
export FINCH_IMAGE="birdhouse/finch:version-${FINCH_VERSION}"
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
export FINCH_IMAGE_URI="registry.hub.docker.com/${FINCH_IMAGE}"
fmigneault marked this conversation as resolved.
Show resolved Hide resolved

# This tells the postgres service to create a database on startup
export POSTGRES_DATABASES_TO_CREATE="
Expand All @@ -28,4 +29,6 @@ COMPONENT_DEPENDENCIES="
OPTIONAL_VARS="
$OPTIONAL_VARS
\$FINCH_VERSION
\$FINCH_IMAGE
\$FINCH_IMAGE_URI
"
9 changes: 9 additions & 0 deletions birdhouse/components/finch/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
"name": "finch",
"version": "${FINCH_VERSION}",
"types": [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hasn't been implemented yet in the schema. Adding this won't cause schema validation to fail but do we plan on updating the schema before or after this PR is merged?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the schema gets updated to include it beforehand, then we can bump its version right away. Otherwise, it can be done later. As mentioned, it doesn't cause an issue regardless at the moment since it is not validated.

"wps"
],
"keywords": [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we leave these keywords as is if we're moving this information to "types". What if we borrowed some of the tags and categories from canarie-api instead. So for finch we could do something like:

"keywords": ["Climatology", "Cloud", "Data Slicer", "Climate Indices", ...]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I would like to do this, but until there is a clear-cut version I can refer to, adding these to the keywords will fail schema validation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Try now with version 1.2.0?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to do this in this PR or in a different one?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, I'd prefer to work on this only once.

"service-wps"
],
Expand All @@ -20,6 +24,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/finch?service=WPS&request=GetCapabilities"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${FINCH_IMAGE_URI}"
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just my 2 cents, since the URI, do not start with http, should we name the new field "uri" instead of "href"? "href" would suggest starting with http usually. Ignore if it has to be "href" for compat or other reasons.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"href" is required by the schema so we can't change the name

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A non-http scheme value is a valid URI. It uniquely identifies the resource.
I would prefer to have a docker:// scheme or similar to be more explicit if permitted, but docker does not seem to like it. However, the values provided in the defaults can be used directly (e.g. docker pull 'registry.hub.docker.com/pavics/weaver:5.0.0' works, but http://registry.hub.docker.com/pavics/weaver:5.0.0 doesn't), so I find them more useful although maybe counter-intuitive.

Maybe one way to avoid confusion would be to use index.docker.io/pavics/weaver:5.0.0 instead, since it does not redirect to https://hub.docker.com/ like registry.hub.docker.com does? This alternate location still woks with docker pull 'index.docker.io/pavics/weaver:5.0.0', and it looks slightly more like the type definition. The idea of this link is really to have the exact value that can be plugged in the docker-compose service image field.

}
]
}
2 changes: 2 additions & 0 deletions birdhouse/components/geoserver/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ export GEOSERVER_DOCKER=pavics/geoserver
export GEOSERVER_VERSION=2.22.2
export GEOSERVER_TAGGED=2.22.2-kartoza-build20230226-r7-allow-change-context-root-and-fix-missing-stable-plugins-and-avoid-chown-datadir
export GEOSERVER_IMAGE="${GEOSERVER_DOCKER}:${GEOSERVER_TAGGED}"
export GEOSERVER_IMAGE_URI="registry.hub.docker.com/${GEOSERVER_IMAGE}"

export GEOSERVER_ADMIN_USER="admin"

Expand Down Expand Up @@ -49,5 +50,6 @@ OPTIONAL_VARS="
\$GEOSERVER_VERSION
\$GEOSERVER_TAGGED
\$GEOSERVER_IMAGE
\$GEOSERVER_IMAGE_URI
\$GEOSERVER_SKIP_AUTH_PROXY_INCLUDE
"
12 changes: 12 additions & 0 deletions birdhouse/components/geoserver/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "geoserver",
"version": "${GEOSERVER_VERSION}",
"types": [
"data",
"wms",
"wfs",
"wps"
],
"keywords": [
"data",
"service-wms",
Expand All @@ -18,6 +25,11 @@
"rel": "service-doc",
"type": "text/html",
"href": "https://docs.geoserver.org/"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${GEOSERVER_IMAGE_URI}"
}
]
}
4 changes: 4 additions & 0 deletions birdhouse/components/hummingbird/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
export HUMMINGBIRD_VERSION=0.5_dev
export HUMMINGBIRD_IMAGE=pavics/hummingbird:${HUMMINGBIRD_VERSION}
export HUMMINGBIRD_IMAGE_URI="registry.hub.docker.com/${HUMMINGBIRD_IMAGE}"

# This tells the postgres service to create a database on startup
export POSTGRES_DATABASES_TO_CREATE="
Expand All @@ -22,4 +24,6 @@ COMPONENT_DEPENDENCIES="
OPTIONAL_VARS="
$OPTIONAL_VARS
\$HUMMINGBIRD_VERSION
\$HUMMINGBIRD_IMAGE
\$HUMMINGBIRD_IMAGE_VERSION
"
2 changes: 1 addition & 1 deletion birdhouse/components/hummingbird/docker-compose-extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ x-logging:

services:
hummingbird:
image: pavics/hummingbird:${HUMMINGBIRD_VERSION}
image: ${HUMMINGBIRD_IMAGE}
container_name: hummingbird
environment:
HOSTNAME: $HOSTNAME
Expand Down
9 changes: 9 additions & 0 deletions birdhouse/components/hummingbird/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "hummingbird",
"version": "${HUMMINGBIRD_VERSION}",
"types": [
"wps"
],
"keywords": [
"service-wps"
],
Expand All @@ -20,6 +24,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/hummingbird?service=WPS&request=GetCapabilities"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${HUMMINGBIRD_IMAGE_URI}"
}
]
}
4 changes: 4 additions & 0 deletions birdhouse/components/jupyterhub/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

export JUPYTERHUB_DOCKER=pavics/jupyterhub
export JUPYTERHUB_VERSION=4.0.2-20240117
export JUPYTERHUB_IMAGE="${JUPYTERHUB_DOCKER}:${JUPYTERHUB_VERSION}"
export JUPYTERHUB_IMAGE_URI="registry.hub.docker.com/${JUPYTERHUB_IMAGE}"

# Jupyter single-user server images, can be overriden in env.local to have a space separated list of multiple images
export DOCKER_NOTEBOOK_IMAGES="pavics/workflow-tests:py39-230601-1-update240116"
Expand Down Expand Up @@ -93,6 +95,8 @@ OPTIONAL_VARS="
\$JUPYTERHUB_CONFIG_OVERRIDE
\$JUPYTERHUB_DOCKER
\$JUPYTERHUB_VERSION
\$JUPYTERHUB_IMAGE
\$JUPYTERHUB_IMAGE_VERSION
\$JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL
\$JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE
\$JUPYTER_IDLE_SERVER_CULL_TIMEOUT
Expand Down
2 changes: 1 addition & 1 deletion birdhouse/components/jupyterhub/docker-compose-extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ x-logging:

services:
jupyterhub:
image: ${JUPYTERHUB_DOCKER}:${JUPYTERHUB_VERSION}
image: ${JUPYTERHUB_IMAGE}
container_name: jupyterhub
hostname: jupyterhub
environment:
Expand Down
9 changes: 9 additions & 0 deletions birdhouse/components/jupyterhub/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "jupyterhub",
"version": "${JUPYTERHUB_VERSION}",
"types": [
"jupyterhub"
],
"keywords": [
"jupyterhub"
],
Expand All @@ -15,6 +19,11 @@
"rel": "service-doc",
"type": "text/html",
"href": "https://jupyter.org/hub"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${JUPYTERHUB_IMAGE_URI}"
}
]
}
4 changes: 4 additions & 0 deletions birdhouse/components/raven/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
export RAVEN_GEO_URL="https://pavics.ouranos.ca/geoserver/"

export RAVEN_VERSION="0.18.1"
export RAVEN_IMAGE="pavics/raven:${RAVEN_VERSION}"
export RAVEN_IMAGE_URI="registry.hub.docker.com/${RAVEN_IMAGE}"

# This tells the postgres service to create a database on startup
export POSTGRES_DATABASES_TO_CREATE="
Expand All @@ -30,4 +32,6 @@ COMPONENT_DEPENDENCIES="
OPTIONAL_VARS="
$OPTIONAL_VARS
\$RAVEN_VERSION
\$RAVEN_IMAGE
\$RAVEN_IMAGE_VERSION
"
2 changes: 1 addition & 1 deletion birdhouse/components/raven/docker-compose-extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ x-logging:

services:
raven:
image: pavics/raven:${RAVEN_VERSION}
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
image: ${RAVEN_IMAGE}
container_name: raven
environment:
PYWPS_CFG: /wps.cfg
Expand Down
9 changes: 9 additions & 0 deletions birdhouse/components/raven/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "raven",
"version": "${RAVEN_VERSION}",
"types": [
"wps"
],
"keywords": [
"service-wps"
],
Expand All @@ -20,6 +24,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/raven?service=WPS&request=GetCapabilities"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${RAVEN_IMAGE_URI}"
}
]
}
24 changes: 24 additions & 0 deletions birdhouse/components/stac/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,20 @@ export STAC_POSTGRES_PASSWORD=${POSTGRES_PAVICS_PASSWORD}
export STAC_PGUSER=${POSTGRES_PAVICS_USERNAME}
export STAC_PGPASSWORD=${POSTGRES_PAVICS_PASSWORD}

# 'main' branch points at https://github.com/stac-utils/stac-fastapi/commit/d53e792
# (see: https://github.com/crim-ca/stac-app/blob/40cad1aa7a094d58fca2d3184182761e248f781d/Dockerfile#L15-L20)
# which corresponds to 2.4.3 release
export STAC_VERSION=2.4.3-crim-main
export STAC_IMAGE=ghcr.io/crim-ca/stac-app:main
export STAC_IMAGE_URI=${STAC_IMAGE}

# 'docker_image_push' branch points at https://github.com/crim-ca/stac-browser/tree/docker_image_push
# which is some commits ahead of 'v3.0.0-beta.5' (and multiple behind latest official releases)
# version name is slightly tweaked to fulfill schema while leaving an obvious trace
export STAC_BROWSER_VERSION=3.0.0-beta.5-crim-docker-image-push
export STAC_BROWSER_IMAGE=ghcr.io/crim-ca/stac-browser:docker_image_push
export STAC_BROWSER_IMAGE_URI=${STAC_BROWSER_IMAGE}

# add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
# single quotes are important in below list to keep variable names intact until 'pavics-compose' parses them
EXTRA_VARS='
Expand All @@ -14,3 +28,13 @@ EXTRA_VARS='
# extend the original 'VARS' from 'birdhouse/pavics-compose.sh' to employ them for template substitution
# adding them to 'VARS', they will also be validated in case of override of 'default.env' using 'env.local'
VARS="$VARS $EXTRA_VARS"

OPTIONAL_VARS="
$OPTIONAL_VARS
\$STAC_VERSION
\$STAC_IMAGE
\$STAC_IMAGE_VERSION
\$STAC_BROWSER_VERSION
\$STAC_BROWSER_IMAGE
\$STAC_BROWSER_IMAGE_URI
"
4 changes: 2 additions & 2 deletions birdhouse/components/stac/docker-compose-extra.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ x-logging:
services:
stac:
container_name: stac
image: ghcr.io/crim-ca/stac-app:main
image: ${STAC_IMAGE}
depends_on:
- stac-db
environment:
Expand All @@ -28,7 +28,7 @@ services:

stac-browser:
container_name: stac-browser
image: ghcr.io/crim-ca/stac-browser:docker_image_push
image: ${STAC_BROWSER_IMAGE}
environment:
- CATALOG_URL=https://${PAVICS_FQDN_PUBLIC}/stac/
- ROOT_PATH=/stac-browser/
Expand Down
20 changes: 20 additions & 0 deletions birdhouse/components/stac/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "stac",
"version": "${STAC_VERSION}",
"types": [
"catalog",
"data"
],
"keywords": [
"catalog"
],
Expand All @@ -25,11 +30,21 @@
"rel": "alternate",
"type": "text/html",
"href": "https://${PAVICS_FQDN_PUBLIC}/stac-browser/"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${STAC_IMAGE_URI}"
}
]
}, {
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "stac-browser",
"version": "${STAC_BROWSER_VERSION}",
"types": [
"catalog",
"data"
],
"keywords": [
"catalog"
],
Expand All @@ -49,6 +64,11 @@
"rel": "alternate",
"type": "application/json",
"href": "https://${PAVICS_FQDN_PUBLIC}/stac/"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${STAC_BROWSER_IMAGE_URI}"
}
]
}
3 changes: 3 additions & 0 deletions birdhouse/components/thredds/default.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
export THREDDS_VERSION=4.6.18-unidata-2022-01
export THREDDS_DOCKER=pavics/thredds-docker
export THREDDS_IMAGE="${THREDDS_DOCKER}:${THREDDS_VERSION}"
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
export THREDDS_IMAGE_URI="registry.hub.docker.com/${THREDDS_IMAGE}"
fmigneault marked this conversation as resolved.
Show resolved Hide resolved
export THREDDS_ORGANIZATION="Birdhouse"
export THREDDS_ADDITIONAL_CATALOG=""

Expand Down Expand Up @@ -37,6 +38,8 @@ OPTIONAL_VARS="
\$TWITCHER_PROTECTED_PATH
\$THREDDS_DOCKER
\$THREDDS_VERSION
\$THREDDS_IMAGE
\$THREDDS_IMAGE_URI
\$THREDDS_ADDITIONAL_CATALOG
"

Expand Down
10 changes: 10 additions & 0 deletions birdhouse/components/thredds/service-config.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
{
"$schema": "https://raw.githubusercontent.com/DACCS-Climate/Marble-node-registry/main/node_registry.schema.json#service",
"name": "thredds",
"version": "${THREDDS_VERSION}",
"types": [
"data",
"catalog"
],
"keywords": [
"data",
"catalog"
Expand All @@ -21,6 +26,11 @@
"rel": "service-desc",
"type": "text/xml",
"href": "https://${PAVICS_FQDN_PUBLIC}/thredds/catalog.xml"
},
{
"rel": "service-meta",
"type": "application/vnd.oci.image.index.v1+json",
"href": "${THREDDS_IMAGE_URI}"
}
]
}
Loading
Loading