Implement hatch (#21)

* commit to hatch * fix files but it doesnt match pre-commit * update config * more reformatting * fixing mypy * fixing types * fix linting, still some pending ignored errors * fix testing errors * type fixing, hatch updates * more hatch setup * fix test command in ci * fix test command in ci * update contributing doc * remove ard in favor of #28 * move around hatch envs * fix CI * Update CONTRIBUTING.md * fix merge conflict issues * pre-commit settings * fix pre-commit * split up pre-commit linting * remove extra version declaration * remove ruff, reorganize * fux build env * remove ruff * move around how env are defined so we can drop into a shell * dep cleanup * remove extra line * review changes * centralize python/hatch install * fix action name * fix step name
dbt-labs · Jan 26, 2024 · 67e25a8 · 67e25a8
1 parent 0910ca4
commit 67e25a8
Show file tree

Hide file tree

Showing 77 changed files with 1,988 additions and 509 deletions.
diff --git a/.flake8 b/.flake8
diff --git a/.github/actions/setup-python-hatch/action.yml b/.github/actions/setup-python-hatch/action.yml
@@ -0,0 +1,20 @@
+name: Setup Python env
+description: Install Python & Hatch
+inputs:
+    python-version:
+      description: 'Version of Python to Install'
+      required: true
+      default: '3.9'
+runs:
+  using: "composite"
+  steps:
+  - name: "Set up Python ${{ inputs.python-version }}"
+    uses: actions/setup-python@v4
+    with:
+      python-version: "${{ inputs.python-version }}"
+
+  - name: Install Hatch
+    shell: bash
+    run: |
+      python -m pip install --user --upgrade pip
+      python -m pip install hatch
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -42,50 +42,14 @@ jobs:
       - name: Check out the repository
         uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v5
+      - name: "Set up Python & Hatch - 3.11"
+        uses: ./.github/actions/setup-python-hatch
         with:
-          python-version: '3.11'
-
-      - name: "Install build specific python dependencies"
-        run: |
-          python -m pip install --user --upgrade pip
-          python -m pip install --upgrade wheel twine check-wheel-contents
-          python -m pip --version
-
-      - name: "Install Hatch"
-        shell: bash
-        run: pip3 install hatch
+          python-version: "3.11"
 
       - name: "Build Python Package"
         run: |
           hatch build
 
-      - name: "Show distributions"
-        run: ls -lh dist/
-
-      - name: "Check distribution descriptions"
-        run: |
-          twine check dist/*
-
-      - name: "Check wheel contents"
-        run: |
-          check-wheel-contents dist/*.whl --ignore W007,W008
-
-      - name: "Install wheel distributions"
-        run: |
-          find ./dist/dbt_common-*.whl -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
-
-      # TODO: how to validate here?  we did dbt --version previously. this checks it's there, but not that it can do anything. maybe it's enough?
-      - name: "Check wheel distributions"
-        run: |
-          pip freeze | grep dbt-common
-
-      - name: "Install source distributions"
-        run: |
-          find ./dist/dbt_common-*.gz -maxdepth 1 -type f | xargs python -m pip install --force-reinstall --find-links=dist/
-
-      # TODO: how to validate here?  we did dbt --version previously. this checks it's there, but not that it can do anything. maybe it's enough?
-      - name: "Check source distributions"
-        run: |
-          pip freeze | grep dbt-common
+      - name: "Check build"
+        run: hatch run build:check-all
diff --git a/.github/workflows/ci_code_quality.yml b/.github/workflows/ci_code_quality.yml
@@ -42,14 +42,14 @@ jobs:
       - name: Check out the repository
         uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v5
+      - name: "Set up Python & Hatch - 3.11"
+        uses: ./.github/actions/setup-python-hatch
         with:
-          python-version: '3.11'
+          python-version: "3.11"
 
-      - name: Install Hatch
+      - name: Install pre-commit
         shell: bash
-        run: pip3 install hatch
+        run: pip3 install pre-commit
 
       - name: Run Pre-commit Hooks
-        run: hatch run dev-env:pre-commit run --show-diff-on-failure --color=always --all-files
+        run: pre-commit run --show-diff-on-failure --color=always --all-files
diff --git a/.github/workflows/ci_tests.yml b/.github/workflows/ci_tests.yml
@@ -47,17 +47,13 @@ jobs:
       - name: "Check out the repository"
         uses: actions/checkout@v4
 
-      - name: "Set up Python ${{ matrix.python-version }}"
-        uses: actions/setup-python@v5
+      - name: "Set up Python & Hatch - ${{ matrix.python-version }}"
+        uses: ./.github/actions/setup-python-hatch
         with:
           python-version: "${{ matrix.python-version }}"
 
-      - name: "Install Hatch"
-        shell: bash
-        run: pip3 install hatch
-
       - name: "Run Tests"
-        run: hatch run dev-env:pytest tests
+        run: hatch run test:unit
 
       - name: "Get current date"
         if: always()

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -10,51 +10,34 @@ repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v3.2.0
   hooks:
+  - id: debug-statements
   - id: check-yaml
     args: [--unsafe]
   - id: end-of-file-fixer
   - id: trailing-whitespace
     exclude_types:
       - "markdown"
   - id: check-case-conflict
-- repo: https://github.com/psf/black
-  rev: 22.3.0
+- repo: local
   hooks:
   - id: black
-  - id: black
-    alias: black-check
-    stages: [manual]
-    args:
-    - "--check"
-    - "--diff"
-- repo: https://github.com/pycqa/flake8
-  rev: 4.0.1
-  hooks:
+    name: black
+    entry: hatch run lint:black
+    language: system
+    types: [python]
+    pass_filenames: false
+    verbose: true
   - id: flake8
-  - id: flake8
-    alias: flake8-check
-    stages: [manual]
-# - repo: https://github.com/pre-commit/mirrors-mypy
-#   rev: v1.4.1
-#   hooks:
-#   - id: mypy
-#     # N.B.: Mypy is... a bit fragile.
-#     #
-#     # By using `language: system` we run this hook in the local
-#     # environment instead of a pre-commit isolated one.  This is needed
-#     # to ensure mypy correctly parses the project.
-
-#     # It may cause trouble
-#     # in that it adds environmental variables out of our control to the
-#     # mix.  Unfortunately, there's nothing we can do about per pre-commit's
-#     # author.
-#     # See https://github.com/pre-commit/pre-commit/issues/730 for details.
-#     args: [--show-error-codes]
-#     files: ^dbt_common/
-#     language: system
-#   - id: mypy
-#     alias: mypy-check
-#     stages: [manual]
-#     args: [--show-error-codes, --pretty]
-#     files: ^dbt_common
-#     language: system
+    name: flake8
+    entry: hatch run lint:flake8
+    language: system
+    types: [python]
+    pass_filenames: false
+    verbose: true
+  - id: mypy
+    name: mypy
+    entry: hatch run lint:mypy
+    language: system
+    types: [python]
+    pass_filenames: false
+    verbose: true
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -54,6 +54,7 @@ There are some tools that will be helpful to you in developing locally. While th
 
 These are the tools used in `dbt-common` development and testing:
 
+- [`hatch`](https://hatch.pypa.io/latest/) for project management
 - [`flake8`](https://flake8.pycqa.org/en/latest/) for code linting
 - [`black`](https://github.com/psf/black) for code formatting
 - [`mypy`](https://mypy.readthedocs.io/en/stable/) for static type checking
@@ -62,31 +63,15 @@ These are the tools used in `dbt-common` development and testing:
 
 A deep understanding of these tools in not required to effectively contribute to `dbt-common`, but we recommend checking out the attached documentation if you're interested in learning more about each one.
 
-#### Virtual environments
-
-We strongly recommend using virtual environments when developing code in `dbt-common`. We recommend creating this virtualenv
-in the root of the `dbt-common` repository. To create a new virtualenv, run:
-```sh
-python3 -m venv env
-source env/bin/activate
-```
-
-This will create and activate a new Python virtual environment.
-
 ## Running `dbt-common` in development
 
 ### Installation
 
-First make sure that you set up your `virtualenv` as described in [Setting up an environment](#setting-up-an-environment).  Also ensure you have the latest version of pip installed with `pip install --upgrade pip`. Next, install `dbt-common` (and its dependencies):
-
-```sh
-git
-pre-commit install
-```
+Ensure you have the latest version of pip installed with `pip install --upgrade pip` as well as [hatch](https://hatch.pypa.io/latest/install/).
 
 ### Running `dbt-common`
 
-This repository is just a template and cannot be run.
+This repository cannot be run on its own.
 
 ## Testing
 
@@ -98,29 +83,31 @@ Once you're able to manually test that your code change is working as expected,
 
 ### Initial setup
 
-None needed.
-
-### Test commands
+- [Install pre-commit](https://pre-commit.com/#usage)
+- [Install hatch](https://hatch.pypa.io/1.7/install/#pip)
 
-No tests included.
+- Nothing needed to set up your environments.  hatch will create your environment as defined in the `pyproject.toml` when you run.
 
+### Hatch Commands
 
-### Unit, Integration, Functional?
+See the pyproject.toml for a complete list of custom commands.  See the h[atch docs](https://hatch.pypa.io/latest/cli/reference/) for a description of built in commands and flags.
 
-Here are some general rules for adding tests:
-* unit tests (`tests/unit`) don’t need to access a database; "pure Python" tests should be written as unit tests
-* functional tests (`tests/functional`) cover anything that interacts with a database, namely adapter
+Run `hatch env show` to view a list of all envoronments and all commands available within them.
 
-## Debugging
+Example uses:
 
-1. The logs for a `dbt run` have stack traces and other information for debugging errors (in `logs/dbt.log` in your project directory).
-2. Try using a debugger, like `ipdb`. For pytest: `--pdb --pdbcls=IPython.terminal.debugger:pdb`
-3. 
+|Type|Command|Description|
+|---|---|---|
+|Utility|`hatch run proto`|regenerate protobuf definitions|
+|Testing|`hatch run test:unit`|run all tests|
+|Code Quality|`hatch run lint:all`|run black, flake8 and mypy checks|
+|Code Quality|`hatch run lint:black`|run black|
+|Shell|`hatch shell`|Drops you into the default shell with  project + dev requirements installed.  Use `exit` to leave the shell, _not_ `deactivate`.|
+|Shell|`hatch -e <environment-name> shell`|Drops you into a shell of the specified environment.  Use `exit` to leave the shell, _not_ `deactivate`.|
 
 ### Assorted development tips
 * Append `# type: ignore` to the end of a line if you need to disable `mypy` on that line.
 * Sometimes flake8 complains about lines that are actually fine, in which case you can put a comment on the line such as: # noqa or # noqa: ANNN, where ANNN is the error code that flake8 issues.
-* To collect output for `CProfile`, run dbt with the `-r` option and the name of an output file, i.e. `dbt -r dbt.cprof run`. If you just want to profile parsing, you can do: `dbt -r dbt.cprof parse`. `pip` install `snakeviz` to view the output. Run `snakeviz dbt.cprof` and output will be rendered in a browser window.
 
 ## Adding or modifying a CHANGELOG Entry
 

diff --git a/Makefile b/Makefile