Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Honeycomb.io sqlfluff workflow #28

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions menu_of_workflows/honeycomb.io/.sqlfluff
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
[sqlfluff]
templater = jinja
dialect = snowflake
max_line_length = 160

[sqlfluff:indentation]
indent_unit = space
tab_space_size = 4

[sqlfluff:rules:layout.long_lines]
ignore_comment_lines = True
ignore_comment_clauses = True

[sqlfluff:rules:capitalisation.keywords]
# for Keywords
capitalisation_policy = lower

[sqlfluff:rules:ambiguous.column_references]
# GROUP BY/ORDER BY column references
group_by_and_order_by_style = explicit

[sqlfluff:rules:convention.blocked_words]
# Comma separated list of blocked words that should not be used
blocked_words = "iff"
blocked_regex = None

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

[sqlfluff:templater:jinja:context]
database=analytics
schema=prod
19 changes: 19 additions & 0 deletions menu_of_workflows/honeycomb.io/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# honeycomb.io SQLFluff GitHub Workflow

This workflow is a massively simplified and updated version of the [surfline workflow](menu_of_workflows/surfline).

To add this to your repo, copy the contents of [`sqlfluff_lint_dbt_models.yml`](./sqlfluff_lint_dbt_models.yml)
in this folder into a file named `.github/workflows/sqlfluff_lint_dbt_models.yml`.

## This GitHub Workflow
- Lints any added or modified models in `/models`, with one exclusion example
- Uses a bare `requirements.txt` file to manage python dependencies. The contents of it are simply `sqlfluff==2.3.5`
- Identifies only the changed files on Pull Requests so that only they are linted.
- Annotates failures on the PR, on the line where they occur, using SQLFluff's native annotation mode.

__NOTE:__ This workflow has been tested on Snowflake, but it should be possible to modify the flow to work with other data warehouses.

## Setup
### `.sqlfluff`

The `.sqlfluff` found [here](./.sqlfluff) skips the dbt templater in favor of the much simpler and faster jinja templater. A few jinja contexts are supplied as an example, and these are typically enough to keep the templater happy.
48 changes: 48 additions & 0 deletions menu_of_workflows/honeycomb.io/sqlfluff_lint_dbt_models.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Original source: https://github.com/sqlfluff/sqlfluff-github-actions/blob/main/menu_of_workflows/surfline/sqlfluff_lint_dbt_models.yml
name: SQLFluff lint dbt data models

on:
push:
branches:
- main
pull_request:
branches:
- main

jobs:
sqlfluff-lint-models:
name: Lint dbt models using SQLFluff
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: ['ubuntu-latest']
python-version: ['3.11']

steps:
- name: Checkout branch
uses: actions/checkout@v3
with:
fetch-depth: 0 # OR "2" -> To retrieve the preceding commit.

- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'

- name: Install Python packages or restore cache
run: pip install -r requirements.txt

- name: Get all changed model SQL files
id: changed-files-models
uses: tj-actions/changed-files@v37
with:
files: models/**/*.sql
files_ignore: models/utils/date_spine.sql

- name: Lint dbt models
if: steps.changed-files-models.outputs.any_changed == 'true'
shell: bash -l {0}
run: |
sqlfluff lint --format github-annotation-native --annotation-level failure ${{ steps.changed-files-models.outputs.all_changed_files }}