Skip to content

Commit

Permalink
update to align with other docker release pipelines
Browse files Browse the repository at this point in the history
  • Loading branch information
mikealfare committed Apr 13, 2024
1 parent acf410d commit 938fc07
Show file tree
Hide file tree
Showing 10 changed files with 114 additions and 60 deletions.
14 changes: 4 additions & 10 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ jobs:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
outputs:
version: ${{ steps.release-inputs.outputs.version }}
is-docker-release: ${{ steps.semver.outputs.is-pre-release == 0 }}

steps:
- name: Check out repository
Expand All @@ -48,12 +47,6 @@ jobs:
echo "archive-name=$archive_name" >> $GITHUB_OUTPUT
echo "version=version" >> $GITHUB_OUTPUT
- name: Audit version to determine if it is a pre-release
id: semver
uses: dbt-labs/actions/[email protected]
with:
version: ${{ steps.release-inputs.outputs.version }}

- name: Build `dbt-postgres`
uses: dbt-labs/dbt-adapters/.github/actions/build-hatch@main
with:
Expand All @@ -68,10 +61,11 @@ jobs:
docker-release:
name: "Docker Release"
needs: [release]
if: ${{ needs.release.outputs.is-docker-release }}
if: ${{ !failure() && !cancelled() }}
permissions:
packages: write
uses: dbt-labs/dbt-release/.github/workflows/release-docker.yml@main
uses: dbt-labs/dbt-release/.github/workflows/release-docker.yml@docker-release
with:
package: "dbt-postgres"
package: ${{ github.event.repository.name }}
version_number: ${{ needs.release.outputs.version }}
test_run: ${{ inputs.deploy-to == 'test' }}
30 changes: 17 additions & 13 deletions dbt/adapters/postgres/relation.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,23 @@

@dataclass(frozen=True, eq=False, repr=False)
class PostgresRelation(BaseRelation):
renameable_relations: FrozenSet[RelationType] = field(default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
RelationType.MaterializedView,
}
))
replaceable_relations: FrozenSet[RelationType] = field(default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
}
))
renameable_relations: FrozenSet[RelationType] = field(
default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
RelationType.MaterializedView,
}
)
)
replaceable_relations: FrozenSet[RelationType] = field(
default_factory=lambda: frozenset(
{
RelationType.View,
RelationType.Table,
}
)
)

def __post_init__(self):
# Check for length of Postgres table/view names.
Expand Down
34 changes: 19 additions & 15 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,20 +1,18 @@
ARG build_for=linux/amd64
# this image gets published to GHCR for production use
ARG py_version=3.10.7

FROM --platform=$build_for python:3.10.7-slim-bullseye as base

# ref is updated automatically every final release via bumpversion
ARG [email protected]
FROM python:$py_version-slim-bullseye as base

RUN apt-get update \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
git \
ssh-client \
software-properties-common \
make \
build-essential \
ca-certificates \
libpq-dev \
build-essential=12.9 \
ca-certificates=20210119 \
git=1:2.30.2-1+deb11u2 \
libpq-dev=13.14-0+deb11u1 \
make=4.3-4.1 \
openssh-client=1:8.4p1-5+deb11u3 \
software-properties-common=0.96.20.2-2.1 \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
Expand All @@ -24,10 +22,16 @@ RUN apt-get update \
ENV PYTHONIOENCODING=utf-8
ENV LANG=C.UTF-8

RUN python -m pip install --upgrade pip setuptools wheel --no-cache-dir
RUN python -m pip install --upgrade "pip==24.0" "setuptools==69.2.0" "wheel==0.43.0" --no-cache-dir


FROM base as dbt-postgres

Check failure on line 28 in docker/Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

Missing User Instruction

Rule ID: e54afcf9-dc71-484a-8967-d930e3044062 Severity: High Resource: FROM={{base as dbt-postgres}} A user should be specified in the dockerfile, otherwise the image will run as root
Raw output
Expected: The 'Dockerfile' should contain the 'USER' instruction
Found: The 'Dockerfile' does not contain any 'USER' instruction

ARG commit_ref=main

HEALTHCHECK CMD dbt --version || exit 1

WORKDIR /usr/app/dbt/
ENTRYPOINT ["dbt"]

FROM base as dbt-postgres
RUN python -m pip install --no-cache-dir "dbt-postgres @ git+https://github.com/dbt-labs/${dbt_postgres_ref}"
RUN python -m pip install --no-cache-dir "dbt-postgres @ git+https://github.com/dbt-labs/dbt-postgres@${commit_ref}"

Check warning on line 37 in docker/Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

Unpinned Package Version in Pip Install

Rule ID: 1f0d05d7-8caf-4f04-bc60-332d472de5a9 Severity: Medium Resource: FROM={{base as dbt-postgres}}.{{RUN python -m pip install --no-cache-dir "dbt-postgres @ git+https://github.com/dbt-labs/dbt-postgres@${commit_ref}"}} Package version pinning reduces the range of versions that can be installed, reducing the chances of failure due to unanticipated changes
Raw output
Expected: RUN instruction with 'pip/pip3 install <package>' should use package pinning form 'pip/pip3 install <package>=<version>'
Found: RUN instruction python -m pip install --no-cache-dir "dbt-postgres @ git+https://github.com/dbt-labs/dbt-postgres@main" does not use package pinning form
20 changes: 4 additions & 16 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@ docker build --tag <your_image_name> --target dbt-postgres <path/to/dockerfile>
---

By default the images will be populated with the most recent release of `dbt-postgres`. If you need to use a different version you can specify it by git ref using the `--build-arg` flag:
By default the image will be populated with the latest version of `dbt-postgres` on `main`.
If you need to use a different version you can specify it by git ref using the `--build-arg` flag:
```shell
docker build --tag <your_image_name> \
--target dbt-postgres \
--build-arg dbt_postgres_ref=<git_ref> \
--build-arg commit_ref=<commit_ref> \
<path/to/dockerfile>
```

Expand All @@ -35,23 +36,10 @@ cd dbt-core/docker
docker build \
--tag my-other-dbt \
--target dbt-postgres \
--build-arg dbt_postgres_ref=dbt-postgres@v1.0.0b1 \
--build-arg commit_ref=v1.0.0b1 \
.
```

## Special cases
There are a few special cases worth noting:

* If you need to build against another architecture (linux/arm64 in this example) you can override the `build_for` build arg:
```shell
docker build --tag my_dbt \
--target dbt-postgres \
--build-arg build_for=linux/arm64 \
<path/to/dockerfile>
```

Supported architectures can be found in the python docker [dockerhub page](https://hub.docker.com/_/python).

## Running an image in a container:
The `ENTRYPOINT` for this Dockerfile is the command `dbt` so you can bind-mount your project to `/usr/app` and use dbt as normal:
```shell
Expand Down
54 changes: 54 additions & 0 deletions docker/dev.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# this image does not get published, it is intended for local development only, see `Makefile` for usage
FROM ubuntu:22.04 as base

# prevent python installation from asking for time zone region
ARG DEBIAN_FRONTEND=noninteractive

# add python repository
RUN apt-get update \

Check notice on line 8 in docker/dev.Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

APT-GET Not Avoiding Additional Packages

Rule ID: 0cbafd91-7f35-4000-b40a-bebedb7bb5f8 Severity: None Resource: FROM={{ubuntu:22.04 as base}}.{{RUN apt-get update && apt-get install -y software-properties-common=0.99.22.9 && add-apt-repository -y ppa:deadsnakes/ppa && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*}} Check if any apt-get installs don't use '--no-install-recommends' flag to avoid installing additional packages.
Raw output
Expected: 'RUN apt-get update   && apt-get install -y software-properties-common=0.99.22.9   && add-apt-repository -y ppa:deadsnakes/ppa   && apt-get clean   && rm -rf     /var/lib/apt/lists/*     /tmp/*     /var/tmp/*' uses '--no-install-recommends' flag to avoid installing additional packages
Found: 'RUN apt-get update   && apt-get install -y software-properties-common=0.99.22.9   && add-apt-repository -y ppa:deadsnakes/ppa   && apt-get clean   && rm -rf     /var/lib/apt/lists/*     /tmp/*     /var/tmp/*' does not use '--no-install-recommends' flag to avoid installing additional packages
&& apt-get install -y software-properties-common=0.99.22.9 \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*

# install python
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential=12.9ubuntu3 \
git-all=1:2.34.1-1ubuntu1.10 \
libpq-dev=14.11-0ubuntu0.22.04.1 \
python3.8=3.8.19-1+jammy1 \
python3.8-dev=3.8.19-1+jammy1 \
python3.8-distutils=3.8.19-1+jammy1 \
python3.8-venv=3.8.19-1+jammy1 \
python3-pip=22.0.2+dfsg-1ubuntu0.4 \
python3-wheel=0.37.1-2ubuntu0.22.04.1 \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*

# update the default system interpreter to the newly installed version
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1

# install python dependencies
RUN python3 -m pip install --upgrade --no-cache-dir "hatch==1.9.1"


FROM base as dbt-postgres-dev

Check failure on line 42 in docker/dev.Dockerfile

View check run for this annotation

Wiz Inc. (266a8a9c32) / Wiz IaC Scanner

Missing User Instruction

Rule ID: e54afcf9-dc71-484a-8967-d930e3044062 Severity: High Resource: FROM={{base as dbt-postgres-dev}} A user should be specified in the dockerfile, otherwise the image will run as root
Raw output
Expected: The 'Dockerfile' should contain the 'USER' instruction
Found: The 'Dockerfile' does not contain any 'USER' instruction

HEALTHCHECK CMD python3 --version || exit 1

# send stdout/stderr to terminal
ENV PYTHONUNBUFFERED=1

# setup mount for local code
WORKDIR /opt/code
VOLUME /opt/code

# create a virtual environment
RUN python3 -m venv /opt/venv
2 changes: 1 addition & 1 deletion docker/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,6 @@ clear \
"#########################################\n"\
&& docker build --tag dbt-postgres-1.0.0b1 \
--target dbt-postgres \
--build-arg dbt_postgres_ref=dbt-postgres@v1.0.0b1 \
--build-arg commit_ref=v1.0.0b1 \
docker \
&& docker run dbt-postgres-1.0.0b1 --version
6 changes: 6 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ dependencies = [
"dbt-adapters @ git+https://github.com/dbt-labs/dbt-adapters.git",
"dbt-common @ git+https://github.com/dbt-labs/dbt-common.git",
]
[tool.hatch.envs.default.scripts]
docker-dev = [
"docker build -f docker/dev.Dockerfile -t dbt-postgres-dev .",
"docker run --rm -it --name dbt-postgres-dev -v $(pwd):/opt/code dbt-postgres-dev",
]
docker-prod = "docker build -f docker/Dockerfile -t dbt-postgres ."

[tool.hatch.envs.lint]
detached = true
Expand Down
1 change: 1 addition & 0 deletions tests/functional/shared_tests/test_hooks/test_hooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
This file needs to be in its own directory because it uses a `data` directory.
Placing this file in its own directory avoids collisions.
"""

from dbt.tests.adapter.hooks.test_model_hooks import (
BasePrePostModelHooks,
BaseHookRefs,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
This file needs to be in its own directory because it creates a `data` directory at run time.
Placing this file in its own directory avoids collisions.
"""

from dbt.tests.adapter.simple_seed.test_seed import (
BaseBasicSeedTests,
BaseSeedConfigFullRefreshOn,
Expand Down
12 changes: 7 additions & 5 deletions tests/unit/test_renamed_relations.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ def test_renameable_relation():
identifier="my_table",
type=RelationType.Table,
)
assert relation.renameable_relations == frozenset({
RelationType.View,
RelationType.Table,
RelationType.MaterializedView,
})
assert relation.renameable_relations == frozenset(
{
RelationType.View,
RelationType.Table,
RelationType.MaterializedView,
}
)

0 comments on commit 938fc07

Please sign in to comment.