Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDDI base Docker build CKAN 2.10.3 #55

Closed
wants to merge 5 commits into from
Closed

SDDI base Docker build CKAN 2.10.3 #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0655a4
SDDI base Docker build CKAN 2.10.3
ilchebedelovski Jan 29, 2024
606e091
Dockerfile 2.10.3 Ubuntu slim based
ilchebedelovski Feb 1, 2024
2c1b4f6
CKAN 2.10.3 build and supervisor enable
ilchebedelovski Feb 3, 2024
25c4519
Dockerfile dependencies
ilchebedelovski Feb 3, 2024
fee51b2
SDDI base Docker build CKAN 2.11.0
aleksandralazoroska Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
294 changes: 238 additions & 56 deletions sddi-base/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,101 @@
###############################################################################
# Build stage
# CKAN build stage
###############################################################################
ARG CKAN_VERSION_BUILD_STAGE=2.9.9-dev
ARG CKAN_VERSION_BUILD_SPATIAL=2.9.9-focal
ARG CKAN_VERSION_RUNTIME_STAGE=2.9.9-focal
FROM python:3.9-slim as ckanbuild

# Used by Github Actions to tag the image with
ENV IMAGE_TAG=2.11.0

# Set CKAN version to build
ENV GIT_URL=https://github.com/ckan/ckan.git
ENV GIT_BRANCH=ckan-2.11.0

# Set src dirs
ENV SRC_DIR=/srv/app/src
ENV PIP_SRC=${SRC_DIR}

WORKDIR ${SRC_DIR}

RUN apt-get update && apt-get install -y \
git \
curl \
libpq-dev \
gcc \
make \
g++ \
autoconf \
automake \
libtool \
patch \
musl-dev \
libpcre3-dev \
libpcre3 \
libffi-dev \
libxml2-dev \
libxslt-dev

# Cleanup to reduce image size
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Link python to python3
RUN ln -s /usr/bin/python3 /usr/bin/python

# Create the src directory
RUN mkdir -p ${SRC_DIR}

# Downgrade setuptools so that CKAN requirements can be built
RUN pip3 install setuptools==44.1.0

# Fetch and build CKAN and requirements
RUN pip3 install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan

# Copy patches and apply patches script
COPY ./patches ${SRC_DIR}/patches
COPY ./scripts/apply_ckan_patches.sh ${SRC_DIR}/apply_ckan_patches.sh

# Apply patches
RUN cd ${SRC_DIR} && ls -lah ${SRC_DIR} && bash ${SRC_DIR}/apply_ckan_patches.sh
RUN rm -rf /srv/app/src/ckan/.git

# Create a constraint file that limits the Cython version to a compatible one, see https://github.com/yaml/pyyaml/issues/736
RUN echo 'Cython < 3.0' > /tmp/constraint.txt
RUN pip3_CONSTRAINT=/tmp/constraint.txt pip3 wheel --wheel-dir=/wheels PyYAML==5.4.1

# RUN pip3-compile ckan/requirements.in
RUN pip3 wheel --wheel-dir=/wheels -r ckan/requirements.txt
RUN pip3 wheel --wheel-dir=/wheels uWSGI==2.0.20 gevent==21.12.0 greenlet==1.1.3

FROM ckan/ckan-base:${CKAN_VERSION_BUILD_STAGE} as extbuild
###############################################################################
# Extbuild stage
###############################################################################
FROM python:3.9-slim as extbuild

USER root

RUN apt-get update && apt-get install -y \
git \
curl \
libpq-dev \
gcc \
make \
g++ \
autoconf \
automake \
libtool \
patch \
musl-dev \
libpcre3-dev \
libpcre3 \
libffi-dev \
libxml2-dev \
libxslt-dev

RUN pip install -U markupsafe==2.0.1

# Create a constraint file that limits the Cython version to a compatible one, see https://github.com/yaml/pyyaml/issues/736
RUN echo 'Cython < 3.0' > /tmp/constraint.txt
RUN pip3_CONSTRAINT=/tmp/constraint.txt pip3 wheel --wheel-dir=/wheels PyYAML==5.4.1

# ckanext-hierarchy ###########################################################
ARG CKANEXT_HIERARCHY_VERSION="v1.2.0"
ENV CKANEXT_HIERARCHY_VERSION=${CKANEXT_HIERARCHY_VERSION}
Expand All @@ -23,6 +110,13 @@ RUN set -ex && \
curl -o /wheels/ckanext-hierarchy.txt https://raw.githubusercontent.com/ckan/ckanext-hierarchy/${CKANEXT_HIERARCHY_VERSION}/requirements.txt && \
ls -lah /wheels

# ckanext-envvars
ENV ENVVARS_GIT_URL=https://github.com/okfn/ckanext-envvars
ENV ENVVARS_GIT_BRANCH=0.0.2

RUN set -ex && \
pip3 wheel --wheel-dir=/wheels git+${ENVVARS_GIT_URL}@${ENVVARS_GIT_BRANCH}#egg=ckanext-envvars

# ckanext-grouphierarchy ######################################################
ARG CKANEXT_SDDI_VERSION="1.1.3"
ENV CKANEXT_SDDI_VERSION=${CKANEXT_SDDI_VERSION}
Expand Down Expand Up @@ -100,64 +194,117 @@ RUN set -ex && \
pip wheel --wheel-dir=/wheels \
git+${CKANEXT_PASSWORD_POLICY_GITHUB_URL}.git@${CKANEXT_PASSWORD_POLICY_VERSION}#egg=ckanext-password-policy

# ckanext-spatial #############################################################
FROM ghcr.io/keitaroinc/ckan:${CKAN_VERSION_BUILD_SPATIAL} as extbuild-spatial

ARG CKANEXT_SPATIAL_VERSION="c2118b9"
ENV CKANEXT_SPATIAL_VERSION=${CKANEXT_SPATIAL_VERSION}

USER root
# ckanext-harvest ###########################################################
ARG CKANEXT_HARVEST_VERSION="v1.5.6"
ENV CKANEXT_HARVEST_VERSION=${CKANEXT_HARVEST_VERSION}
ENV CKANEXT_HARVEST_GITHUB_URL="https://github.com/ckan/ckanext-harvest.git"

# Install any system packages necessary to build extensions
RUN set -ex && \
apt-get update && \
apt-get install -y --no-install-recommends \
python3-dev python3-pip libxml2-dev libxslt1-dev libgeos-c1v5 python-is-python3 && \
mkdir -p /wheels && \
pip install -U pip
pip install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/${CKANEXT_HARVEST_VERSION}/dev-requirements.txt

RUN set -ex && \
pip install -r https://raw.githubusercontent.com/MarijaKnezevic/ckanext-spatial/${CKANEXT_SPATIAL_VERSION}/requirements.txt && \
curl -o /wheels/ckanext-spatial.txt \
https://raw.githubusercontent.com/MarijaKnezevic/ckanext-spatial/${CKANEXT_SPATIAL_VERSION}/requirements.txt && \
pip install -r https://raw.githubusercontent.com/MarijaKnezevic/ckanext-spatial/${CKANEXT_SPATIAL_VERSION}/requirements-postgis.txt && \
curl -o /wheels/ckanext-spatial-postgis.txt \
https://raw.githubusercontent.com/MarijaKnezevic/ckanext-spatial/${CKANEXT_SPATIAL_VERSION}/requirements-postgis.txt && \
ls -lah /wheels
pip wheel --wheel-dir=/wheels -r https://raw.githubusercontent.com/ckan/ckanext-harvest/${CKANEXT_HARVEST_VERSION}/requirements.txt && \
pip wheel --wheel-dir=/wheels git+https://github.com/ckan/ckanext-harvest.git@${CKANEXT_HARVEST_VERSION}#egg=ckanext-harvest && \
curl -o /wheels/ckanext-harvest.txt https://raw.githubusercontent.com/ckan/ckanext-harvest/${CKANEXT_HARVEST_VERSION}/requirements.txt && \
ls -lah /wheels

# ckanext-spatial #############################################################
ENV CKANEXT_SPATIAL_GITHUB_URL="https://github.com/ckan/ckanext-spatial"
ENV CKANEXT_SPATIAL_VERSION="release-v2.1.0"

RUN set -ex && \
curl -o /wheels/ckanext-spatial-requirements.txt \
https://raw.githubusercontent.com/ckan/ckanext-spatial/${CKANEXT_SPATIAL_VERSION}/requirements-py2.txt && \
pip wheel --wheel-dir=/wheels \
git+https://github.com/MarijaKnezevic/ckanext-spatial.git@${CKANEXT_SPATIAL_VERSION}#egg=ckanext-spatial
git+${CKANEXT_SPATIAL_GITHUB_URL}.git@${CKANEXT_SPATIAL_VERSION}#egg=ckanext-spatial

###############################################################################
# Runtime stage
###############################################################################
FROM ghcr.io/keitaroinc/ckan:${CKAN_VERSION_RUNTIME_STAGE} as runtime

ENV CKAN__PLUGINS "image_view text_view recline_view webpage_view datastore datapusher \
hierarchy_display hierarchy_form display_group relation \
spatial_metadata spatial_query datesearch repeating composite scheming_datasets \
password_policy \
envvars"

# Extra env for compatibility with ckan/base Docker images for downstream k8s
FROM python:3.9-slim

ENV APP_DIR=/srv/app
ENV SRC_DIR=/srv/app/src
ENV CKAN_DIR=${SRC_DIR}/ckan
ENV DATA_DIR=/srv/app/data
ENV PIP_SRC=${SRC_DIR}
ENV GIT_BRANCH=ckan-2.11.0

# Setting the locale
ENV LC_ALL="en_US.UTF-8"
RUN apt-get update && apt-get install --no-install-recommends -y locales
RUN sed -i "/$LC_ALL/s/^# //g" /etc/locale.gen
RUN dpkg-reconfigure --frontend=noninteractive locales
RUN update-locale LANG=${LC_ALL}

# Set timezone
RUN echo "UTC" > /etc/timezone
ENV CKAN_INI=${APP_DIR}/production.ini
ENV CKAN_STORAGE_PATH=/var/lib/ckan
ENV TZ="UTC"

USER root
# Update the package lists and install required packages
RUN apt-get update && apt-get install -y \
bash \
git \
gettext \
curl \
wget \
unzip \
postgresql-client \
libmagic1 \
libpcre3 \
libxslt1.1 \
libxml2 \
tzdata \
apache2-utils \
musl-dev \
libssl-dev \
proj-bin \
libproj-dev \
proj-data \
python3-cffi \
supervisor

# Cleanup to reduce image size
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

RUN pip install markupsafe==2.0.1 setuptools wheel

# Create a constraint file that limits the Cython version to a compatible one, see https://github.com/yaml/pyyaml/issues/736
RUN echo 'Cython < 3.0' > /tmp/constraint.txt
RUN pip3_CONSTRAINT=/tmp/constraint.txt pip3 wheel --wheel-dir=/wheels PyYAML==5.4.1

# Create SRC_DIR
RUN mkdir -p ${SRC_DIR} && \
# Link python to python3
ln -s /usr/bin/python3 /usr/bin/python

# Get artifacts from build stages
COPY --from=ckanbuild /wheels ${APP_DIR}/wheels
COPY --from=extbuild /wheels ${APP_DIR}/ext_wheels
COPY --from=ckanbuild ${APP_DIR}/src/ckan ${CKAN_DIR}

# Install any system packages necessary to build extensions
RUN set -ex && \
apt-get update && \
apt-get install -y --no-install-recommends \
libxml2-dev libxslt1-dev libgeos-c1v5 && \
pip install --no-cache-dir -U pip && \
rm -rf /var/lib/apt/lists/*
# Additional install steps for build stages artifacts
RUN pip3 install --no-index --find-links=${APP_DIR}/wheels uWSGI==2.0.20 gevent==21.12.0

# Copy python wheels from build stage
COPY --from=extbuild /wheels ${APP_DIR}/ext_wheels
COPY --from=extbuild-spatial /wheels ${APP_DIR}/ext_wheels
# Create a local user and group to run the app
# Add a group with a specific GID (92)
RUN groupadd -g 92 ckan
# Add a user with a specific UID (92), home directory, and add to the ckan group
RUN useradd -u 92 -g ckan -M -d ${APP_DIR} -s /bin/bash ckan

WORKDIR ${CKAN_DIR}

# Install CKAN
RUN pip3 install -e ${APP_DIR}/src/ckan
RUN cp who.ini ${APP_DIR}
RUN pip3 install --no-index --find-links=${APP_DIR}/wheels -r requirements.txt

# ckanext-harvest ###########################################################
RUN set -ex && \
pip install --no-index --find-links=${APP_DIR}/ext_wheels ckanext-harvest && \
pip install --no-index --find-links=${APP_DIR}/ext_wheels -r ${APP_DIR}/ext_wheels/ckanext-harvest.txt

# ckanext-hierarchy ###########################################################
RUN set -ex && \
Expand All @@ -169,16 +316,9 @@ RUN set -ex && \
pip install --find-links=${APP_DIR}/ext_wheels -r ${APP_DIR}/ext_wheels/ckanext-grouphierarchy.txt && \
pip install --no-index --find-links=${APP_DIR}/ext_wheels ckanext-grouphierarchy

# ckanext-relation ############################################################
RUN set -ex && \
pip install --find-links=${APP_DIR}/ext_wheels -r ${APP_DIR}/ext_wheels/ckanext-relation.txt && \
pip install --no-index --find-links=${APP_DIR}/ext_wheels ckanext-relation

# ckanext-spatial #############################################################
# ckanext-envvars ############################################################
RUN set -ex && \
pip install -r ${APP_DIR}/ext_wheels/ckanext-spatial.txt && \
pip install -r ${APP_DIR}/ext_wheels/ckanext-spatial-postgis.txt && \
pip install --no-index --find-links=${APP_DIR}/ext_wheels ckanext-spatial
pip install --no-index --find-links=${APP_DIR}/ext_wheels ckanext-envvars

# ckanext-scheming ############################################################
RUN set -ex && \
Expand All @@ -201,10 +341,31 @@ RUN set -ex && \
pip install -r ${APP_DIR}/ext_wheels/ckanext-password-policy.txt && \
pip install --no-index --find-links=${APP_DIR}/ext_wheels ckanext-password-policy

# ckanext-spatial #############################################################
RUN set -ex && \
pip3 install -e 'git+https://github.com/ckan/ckanext-spatial.git#egg=ckanext-spatial' && \
pip3 install -r 'https://raw.githubusercontent.com/ckan/ckanext-spatial/master/requirements.txt'

# Copy init scripts and additional files
COPY --chown=ckan:ckan initScripts/ ${APP_DIR}/docker-afterinit.d
COPY --chown=ckan:ckan who.ini ${APP_DIR}/who.ini

ENV CKAN__PLUGINS "envvars image_view text_view recline_view webpage_view datastore \
harvest ckan_harvester \
hierarchy_display hierarchy_form \
# datapusher Token required \
# display_group ValueError: Cannot determine url for /usr/local/lib/python3.9/site-packages/ckanext/grouphierarchy/fanstatic/hierarchy_theme.css \
# relation ImportError: cannot import name 'Mapping' from 'collections' \
spatial_metadata spatial_query \
scheming_datasets \
datesearch \
composite \
repeating"
# password_policy No module named 'ckan.lib.repoze_plugins' \

RUN set -ex && \
ckan generate config ${APP_DIR}/production.ini

RUN set -ex && \
ckan config-tool "${CKAN_INI}" "ckan.plugins = ${CKAN__PLUGINS}" && \
ckan config-tool "${CKAN_INI}" "ckan.spatial.srid = 4326" && \
Expand All @@ -225,4 +386,25 @@ RUN set -ex && \
# Remove wheels
rm -rf ${APP_DIR}/ext_wheels

USER ckan
WORKDIR ${APP_DIR}

ENV UWSGI_HARAKIRI=50

# Create local storage folder
RUN mkdir -p ${CKAN_STORAGE_PATH} && \
chown -R ckan:ckan ${CKAN_STORAGE_PATH}

# Copy local scripts
COPY setup/prerun.py ${APP_DIR}
COPY setup/start_ckan.sh ${APP_DIR}
ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${APP_DIR}
RUN chmod 644 ${APP_DIR}/wsgi.py

# Create entrypoint directory for children image scripts
ONBUILD RUN mkdir /docker-entrypoint.d

EXPOSE 5000

HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit CMD ["/srv/app/start_ckan.sh"]

CMD ["/srv/app/start_ckan.sh"]
11 changes: 11 additions & 0 deletions sddi-base/patches/00_patch_sql_url.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
--- ckan/ckan/model/__init__.py
+++ ckan/ckan/model/__init__.py
@@ -276,7 +276,7 @@ class Repository():
self.reset_alembic_output()
alembic_config = AlembicConfig(self._alembic_ini)
alembic_config.set_main_option(
- "sqlalchemy.url", config.get("sqlalchemy.url")
+ "sqlalchemy.url", config.get("sqlalchemy.url").replace('%', '%%')
)
try:
sqlalchemy_migrate_version = self.metadata.bind.execute(
10 changes: 10 additions & 0 deletions sddi-base/patches/02_patch_postgres_username_split.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
--- ckan/ckanext/datastore/backend/postgres.py
+++ ckan/ckanext/datastore/backend/postgres.py
@@ -1809,7 +1809,7 @@ class DatastorePostgresqlBackend(DatastoreBackend):
read only user.
'''
write_connection = self._get_write_engine().connect()
- read_connection_user = sa_url.make_url(self.read_url).username
+ read_connection_user = sa_url.make_url(self.read_url).username.split("@")[0]

drop_foo_sql = u'DROP TABLE IF EXISTS _foo'
5 changes: 5 additions & 0 deletions sddi-base/scripts/apply_ckan_patches.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/bin/bash
shopt -s nullglob
for patch in patches/*.patch; do
/usr/bin/patch -p0 -i $patch
done
Loading