[Tech Debt] Managing dev dependencies and dependabot across adapter repos

[mikealfare]

Housekeeping

• I am a maintainer of dbt-adapters

Short description

Design and build a process for managing development dependencies that minimizes effort on support.

Context

We get a lot of dependabot PRs for development requirements, both for being out of date, or for security vulnerabilities. The latter are generally false positives as the development dependencies should not be shipped. Since most dependabot PRs are for development requirements, they tend to get overlooked as unnecessary and become stale.

We want to be able to move pins on development dependencies over time, but not run into issues on older branches due to the lack of a pin. Leaving all development dependencies pinned violates one of these, and leaving them all unpinned violates the other scenario.

We run into issues (e.g. on dbt-redshift) where there are version conflicts between build dependencies and development dependencies that causes development and CI to either take a long time, or fail for reasons other than true test failures. Ideally our virtual environment looks as close as possible to the production build. Adding more dependencies, especially pinned, may inadvertently pin a requirement in development but leave it open in production.

We have scenarios where tools are listed twice with different dependencies because they exist in both .pre-commit-config.yml and dev-requirements.txt.

Since this represents a change in operating model, the right time to do this is along with the move to pyproject.toml and hatch. Otherwise we'll shift folks twice in close succession.

Acceptance criteria

• Manage as many dependencies via .pre-commit-config.yml as possible, specifically including linters and typecheckers
• Minimize dev-requirements.txt to remaining dev dependencies, test dependencies, and build tool dependencies
• Create GHA workflow that runs weekly, updates .pre-commit-config.yml, tests the update, merges the update if successful, and creates a PR if not
• Create an ADR documenting this process
• Update the contributor guide demonstrating how to run linting and typechecking manually

PRs

Preview Give feedback

1. Move to pre-commit only dbt-snowflake#996
   Skip Changelog cla:yes +2
   
2. Move to pre-commit only dbt-spark#1026
   Skip Changelog cla:yes +2
   
3. Move to pre-commit only dbt-redshift#777
   Skip Changelog cla:yes +2
   
4. [Backport 1.7.latest] Move to pre-commit only (#777) dbt-redshift#797
   Skip Changelog cla:yes +2
   
5. [Backport 1.6.latest] Move to pre-commit only (#777) dbt-redshift#798
   Skip Changelog cla:yes +2
   

Testing

• We could leave one of the repos out of date and make sure that the workflow catches it

Security

We're effectively replacing dependabot with our own version. However, we're only doing so for dev dependencies like linters and testers. It might be worth checking in with security to make sure this doesn't violate some company policy that enforces use of dependabot in all scenarios.

Docs

The ADR and contributor guide updates should suffice.

Consequences

Pro's:

• less dependabot PRs
• dependabot PRs gain more attention since they are likely production dependencies now
• dev requirements stay up to date on main
• dev requirements become hard pinned on release branches
• less clutter for support rotation
• more aligned with hatch's environments approach (separate environments for separate tasks)

Con's:

• one more GHA to maintain
• a change to the current operating model (e.g. black is no longer directly runnable)