Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docker release to the full release process for final releases #1004

Merged
merged 30 commits into from
May 21, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3186972
add docker release to release pipeline
mikealfare Apr 5, 2024
446bb0c
changelog
mikealfare Apr 5, 2024
43e100c
Merge branch 'refs/heads/main' into config/docker-release
mikealfare Apr 13, 2024
548f7df
update docker release to align with other adapters, add dev docker
mikealfare Apr 13, 2024
8e31cc4
update docker release to align with other adapters, add dev docker
mikealfare Apr 13, 2024
e9420b5
remove defaulted input for docker package, override default for docke…
mikealfare Apr 15, 2024
acdc453
point back to main
mikealfare Apr 15, 2024
fb197f0
remove changie entry
mikealfare Apr 15, 2024
fc7de15
fix docker release dependent steps
mikealfare Apr 16, 2024
d9d27b0
only release docker when not testing, allow to only release to docker
emmyoop Apr 17, 2024
c685f9d
Merge branch 'refs/heads/main' into config/docker-release
mikealfare May 2, 2024
a82838a
remove dev container
mikealfare May 2, 2024
c1f7359
clean up test script
mikealfare May 2, 2024
1d9fe5d
Update docker-release/Dockerfile
mikealfare May 2, 2024
6c544c6
rename the spark Dockerfile to make space for the release Dockerfile
mikealfare May 2, 2024
52ad6cb
move the release Dockerfile into ./docker
mikealfare May 2, 2024
5f3e52d
move the release Dockerfile into ./docker
mikealfare May 2, 2024
2326b2d
move the release Dockerfile into ./docker
mikealfare May 2, 2024
b87999b
move the release Dockerfile into ./docker
mikealfare May 2, 2024
85c9d9f
Merge branch 'main' into config/docker-release
mikealfare May 3, 2024
28b8aff
Merge branch 'main' into config/docker-release
mikealfare May 7, 2024
d49184f
Merge branch 'main' into config/docker-release
mikealfare May 13, 2024
78b9a36
Merge branch 'main' into config/docker-release
mikealfare May 14, 2024
0680722
Merge branch 'main' into config/docker-release
mikealfare May 14, 2024
dcdab2d
Merge branch 'main' into config/docker-release
mikealfare May 14, 2024
29011fd
Merge branch 'main' into config/docker-release
mikealfare May 14, 2024
4674f9f
point to dev branch for now
mikealfare May 14, 2024
74f8c78
Merge remote-tracking branch 'origin/config/docker-release' into conf…
mikealfare May 14, 2024
c7731c3
point back to main
mikealfare May 15, 2024
562bebd
remove unused script
mikealfare May 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20240405-185609.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Automate the Docker release as part of the PyPI release
time: 2024-04-05T18:56:09.981637-04:00
custom:
Author: mikealfare
Issue: "1004"
16 changes: 15 additions & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,22 @@
version: 2
updates:
# python dependencies
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
rebase-strategy: "disabled"
- package-ecosystem: "docker"
directory: "/docker"
schedule:
interval: "weekly"
rebase-strategy: "disabled"
- package-ecosystem: "docker"
directory: "/docker-dev"
schedule:
interval: "weekly"
rebase-strategy: "disabled"
- package-ecosystem: "docker"
directory: "/docker-release"
schedule:
interval: "weekly"
rebase-strategy: "disabled"
38 changes: 15 additions & 23 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@
# This will only run manually. Run this workflow only after the
# version bump workflow is completed and related changes are reviewed and merged.
#

name: Release to GitHub and PyPI
name: "Release to GitHub, PyPI, and Docker"
run-name: "Release ${{ inputs.version_number }} to GitHub, PyPI, and Docker"

on:
workflow_dispatch:
Expand Down Expand Up @@ -88,26 +88,20 @@ jobs:
# so that the integration tests stage will be skipped.
audit-version-and-changelog:
name: Bump package version, Generate changelog

uses: dbt-labs/dbt-spark/.github/workflows/release-prep.yml@main

with:
sha: ${{ inputs.sha }}
version_number: ${{ inputs.version_number }}
target_branch: ${{ inputs.target_branch }}
env_setup_script_path: ""
test_run: ${{ inputs.test_run }}

secrets: inherit

log-outputs-audit-version-and-changelog:
name: "[Log output] Bump package version, Generate changelog"
if: ${{ !failure() && !cancelled() }}

needs: [audit-version-and-changelog]

runs-on: ubuntu-latest

steps:
- name: Print variables
run: |
Expand All @@ -118,9 +112,7 @@ jobs:
name: Build, Test, Package
if: ${{ !failure() && !cancelled() }}
needs: [audit-version-and-changelog]

uses: dbt-labs/dbt-release/.github/workflows/build.yml@main

with:
sha: ${{ needs.audit-version-and-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
Expand All @@ -129,19 +121,15 @@ jobs:
s3_bucket_name: ${{ inputs.s3_bucket_name }}
package_test_command: ${{ inputs.package_test_command }}
test_run: ${{ inputs.test_run }}

secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

github-release:
name: GitHub Release
if: ${{ !failure() && !cancelled() }}

needs: [audit-version-and-changelog, build-test-package]

uses: dbt-labs/dbt-release/.github/workflows/github-release.yml@main

with:
sha: ${{ needs.audit-version-and-changelog.outputs.final_sha }}
version_number: ${{ inputs.version_number }}
Expand All @@ -150,34 +138,38 @@ jobs:

pypi-release:
name: PyPI Release

needs: [github-release]

uses: dbt-labs/dbt-release/.github/workflows/pypi-release.yml@main

with:
version_number: ${{ inputs.version_number }}
test_run: ${{ inputs.test_run }}

secrets:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
TEST_PYPI_API_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}

docker-release:
name: "Docker Release"
if: ${{ !failure() && !cancelled() }}
needs: [bump-version-generate-changelog, build-test-package, github-release]
permissions:
packages: write
uses: dbt-labs/dbt-release/.github/workflows/release-docker.yml@docker-release
with:
package: ${{ github.event.repository.name }}
version_number: ${{ inputs.version_number }}
test_run: ${{ inputs.test_run }}

slack-notification:
name: Slack Notification
if: ${{ failure() && (!inputs.test_run || inputs.nightly_release) }}

needs:
[
audit-version-and-changelog,
build-test-package,
github-release,
pypi-release,
docker-release,
]

uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"

secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEV_CORE_ALERTS }}
10 changes: 10 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,13 @@ help: ## Show this help message.
@echo
@echo 'targets:'
@grep -E '^[7+a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'


.PHONY: docker-dev
docker-dev:
docker build -f docker-dev/Dockerfile -t dbt-spark-dev .
docker run --rm -it --name dbt-spark-dev -v $(shell pwd):/opt/code dbt-spark-dev

.PHONY: docker-prod
docker-prod:
docker build -f docker-release/Dockerfile -t dbt-spark .
59 changes: 59 additions & 0 deletions docker-dev/Dockerfile
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# this image does not get published, it is intended for local development only, see `Makefile` for usage
FROM ubuntu:22.04 as base

# prevent python installation from asking for time zone region
ARG DEBIAN_FRONTEND=noninteractive

# add python repository
RUN apt-get update \

Check notice on line 8 in docker-dev/Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

APT-GET Not Avoiding Additional Packages

Rule ID: 0cbafd91-7f35-4000-b40a-bebedb7bb5f8 Severity: None Resource: FROM={{ubuntu:22.04 as base}}.{{RUN apt-get update && apt-get install -y software-properties-common=0.99.22.9 && add-apt-repository -y ppa:deadsnakes/ppa && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*}} Check if any apt-get installs don't use '--no-install-recommends' flag to avoid installing additional packages.
Raw output
Expected: 'RUN apt-get update   && apt-get install -y software-properties-common=0.99.22.9   && add-apt-repository -y ppa:deadsnakes/ppa   && apt-get clean   && rm -rf     /var/lib/apt/lists/*     /tmp/*     /var/tmp/*' uses '--no-install-recommends' flag to avoid installing additional packages
Found: 'RUN apt-get update   && apt-get install -y software-properties-common=0.99.22.9   && add-apt-repository -y ppa:deadsnakes/ppa   && apt-get clean   && rm -rf     /var/lib/apt/lists/*     /tmp/*     /var/tmp/*' does not use '--no-install-recommends' flag to avoid installing additional packages
&& apt-get install -y software-properties-common=0.99.22.9 \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*

# install python
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential=12.9ubuntu3 \
gcc=4:11.2.0-1ubuntu1 \
git-all=1:2.34.1-1ubuntu1.10 \
libsasl2-dev=2.1.27+dfsg2-3ubuntu1.2 \
python3.10=3.10.12-1~22.04.3 \
python3.10-dev=3.10.12-1~22.04.3 \
python3.10-venv=3.10.12-1~22.04.3 \
python3-distutils=3.10.8-1~22.04 \
python3-pip=22.0.2+dfsg-1ubuntu0.4 \
python3-wheel=0.37.1-2ubuntu0.22.04.1 \
unixodbc-dev=2.3.9-5ubuntu0.1 \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*

# update the default system interpreter to the newly installed version
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1

# install python dependencies
RUN python3 -m pip install --upgrade --no-cache-dir \
"beartype==0.17.2" \
"dagger-io==0.10.3" \
"python-dotenv==1.0.1"


FROM base as dbt-spark-dev

Check failure on line 47 in docker-dev/Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

Missing User Instruction

Rule ID: e54afcf9-dc71-484a-8967-d930e3044062 Severity: High Resource: FROM={{base as dbt-spark-dev}} A user should be specified in the dockerfile, otherwise the image will run as root
Raw output
Expected: The 'Dockerfile' should contain the 'USER' instruction
Found: The 'Dockerfile' does not contain any 'USER' instruction

HEALTHCHECK CMD python3 --version || exit 1

# send stdout/stderr to terminal
ENV PYTHONUNBUFFERED=1

# setup mount for local code
WORKDIR /opt/code
VOLUME /opt/code

# create a virtual environment
RUN python3 -m venv /opt/venv
42 changes: 42 additions & 0 deletions docker-release/Dockerfile
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# this image gets published to GHCR for production use
ARG py_version=3.10.7
mikealfare marked this conversation as resolved.
Show resolved Hide resolved

FROM python:$py_version-slim-bullseye as base

RUN apt-get update \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
build-essential=12.9 \
ca-certificates=20210119 \
gcc=4:10.2.1-1 \
git=1:2.30.2-1+deb11u2 \
libpq-dev=13.14-0+deb11u1 \
libsasl2-dev=2.1.27+dfsg-2.1+deb11u1 \
make=4.3-4.1 \
openssh-client=1:8.4p1-5+deb11u3 \
python-dev-is-python2=2.7.18-9 \
software-properties-common=0.96.20.2-2.1 \
unixodbc-dev=2.3.6-0.1+b1 \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*

ENV PYTHONIOENCODING=utf-8
ENV LANG=C.UTF-8

RUN python -m pip install --upgrade "pip==24.0" "setuptools==69.2.0" "wheel==0.43.0" --no-cache-dir


FROM base as dbt-spark

Check failure on line 32 in docker-release/Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

Missing User Instruction

Rule ID: e54afcf9-dc71-484a-8967-d930e3044062 Severity: High Resource: FROM={{base as dbt-spark}} A user should be specified in the dockerfile, otherwise the image will run as root
Raw output
Expected: The 'Dockerfile' should contain the 'USER' instruction
Found: The 'Dockerfile' does not contain any 'USER' instruction

ARG commit_ref=main
ARG extras=all

HEALTHCHECK CMD dbt --version || exit 1

WORKDIR /usr/app/dbt/
ENTRYPOINT ["dbt"]

RUN python -m pip install --no-cache-dir "dbt-spark[${extras}] @ git+https://github.com/dbt-labs/dbt-spark@${commit_ref}"

Check warning on line 42 in docker-release/Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

Unpinned Package Version in Pip Install

Rule ID: 1f0d05d7-8caf-4f04-bc60-332d472de5a9 Severity: Medium Resource: FROM={{base as dbt-spark}}.{{RUN python -m pip install --no-cache-dir "dbt-spark[${extras}] @ git+https://github.com/dbt-labs/dbt-spark@${commit_ref}"}} Package version pinning reduces the range of versions that can be installed, reducing the chances of failure due to unanticipated changes
Raw output
Expected: RUN instruction with 'pip/pip3 install <package>' should use package pinning form 'pip/pip3 install <package>=<version>'
Found: RUN instruction python -m pip install --no-cache-dir "dbt-spark[all] @ git+https://github.com/dbt-labs/dbt-spark@main" does not use package pinning form
71 changes: 71 additions & 0 deletions docker-release/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Docker for dbt
This docker file is suitable for building dbt Docker images locally or using with CI/CD to automate populating a container registry.


## Building an image:
This Dockerfile can create images for the following target: `dbt-spark`

In order to build a new image, run the following docker command.
```shell
docker build --tag <your_image_name> --target dbt-spark <path/to/dockerfile>
```
---
> **Note:** Docker must be configured to use [BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/) in order for images to build properly!

---

By default the image will be populated with the latest version of `dbt-spark` on `main`.
If you need to use a different version you can specify it by git ref using the `--build-arg` flag:
```shell
docker build --tag <your_image_name> \
--target dbt-spark \
--build-arg commit_ref=<commit_ref> \
<path/to/dockerfile>
```

### Examples:
To build an image named "my-dbt" that supports Snowflake using the latest releases:
```shell
cd dbt-core/docker
docker build --tag my-dbt --target dbt-spark .
```

To build an image named "my-other-dbt" that supports Snowflake using the adapter version 1.0.0b1:
```shell
cd dbt-core/docker
docker build \
--tag my-other-dbt \
--target dbt-spark \
--build-arg commit_ref=v1.0.0b1 \
.
```

## Special cases
There are a few special cases worth noting:
* The `dbt-spark` database adapter comes in three different versions named `PyHive`, `ODBC`, and the default `all`.
If you wish to override this you can use the `--build-arg` flag with the value of `extras=<extras_name>`.
See the [docs](https://docs.getdbt.com/reference/warehouse-profiles/spark-profile) for more information.
```shell
docker build --tag my_dbt \
--target dbt-spark \
--build-arg commit_ref=v1.0.0b1 \
--build-arg extras=PyHive \
<path/to/dockerfile>
```

## Running an image in a container:
The `ENTRYPOINT` for this Dockerfile is the command `dbt` so you can bind-mount your project to `/usr/app` and use dbt as normal:
```shell
docker run \
--network=host \
--mount type=bind,source=path/to/project,target=/usr/app \
--mount type=bind,source=path/to/profiles.yml,target=/root/.dbt/profiles.yml \
my-dbt \
ls
```
---
**Notes:**
* Bind-mount sources _must_ be an absolute path
* You may need to make adjustments to the docker networking setting depending on the specifics of your data warehouse/database host.

---
22 changes: 22 additions & 0 deletions docker-release/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# - VERY rudimentary test script to run latest + specific branch image builds and test them all by running `--version`
# TODO: create a real test suite

clear \
&& echo "\n\n"\
"########################################\n"\
"##### Testing dbt-spark latest #####\n"\
"########################################\n"\
&& docker build --tag dbt-spark \
--target dbt-spark \
docker \
&& docker run dbt-spark --version \
\
&& echo "\n\n"\
"#########################################\n"\
"##### Testing dbt-spark-1.0.0b1 #####\n"\
"#########################################\n"\
&& docker build --tag dbt-spark-1.0.0b1 \
--target dbt-spark \
--build-arg commit_ref=v1.0.0b1 \
docker \
&& docker run dbt-spark-1.0.0b1 --version
Loading