-
Notifications
You must be signed in to change notification settings - Fork 234
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'origin' into feature/issue_239
- Loading branch information
Showing
86 changed files
with
2,545 additions
and
952 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
## Previous Releases | ||
For information on prior major and minor releases, see their changelogs: | ||
- [1.2](https://github.com/dbt-labs/dbt-spark/blob/1.2.latest/CHANGELOG.md) | ||
- [1.1](https://github.com/dbt-labs/dbt-spark/blob/1.1.latest/CHANGELOG.md) | ||
- [1.0](https://github.com/dbt-labs/dbt-spark/blob/1.0.latest/CHANGELOG.md) | ||
- [0.21](https://github.com/dbt-labs/dbt-spark/blob/0.21.latest/CHANGELOG.md) | ||
- [0.20](https://github.com/dbt-labs/dbt-spark/blob/0.20.latest/CHANGELOG.md) | ||
- [0.19 and earlier](https://github.com/dbt-labs/dbt-spark/blob/0.19.latest/CHANGELOG.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# CHANGELOG | ||
|
||
To view information about the changelog operation we suggest reading this [README](https://github.com/dbt-labs/dbt-spark/blob/main/.changes/README.md) found in `dbt-spark`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# dbt-spark Changelog | ||
|
||
- This file provides a full account of all changes to `dbt-spark`. | ||
- Changes are listed under the (pre)release in which they first appear. Subsequent releases include changes from previous releases. | ||
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version. | ||
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-spark/blob/main/CONTRIBUTING.md#adding-changelog-entry) |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
kind: Breaking Changes | ||
body: Add schema to the default location root | ||
time: 2022-10-31T20:11:09.291461+01:00 | ||
custom: | ||
Author: dan1elt0m JCZuurmond | ||
Issue: "239" | ||
PR: "339" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
kind: Features | ||
body: Migrate dbt-utils current_timestamp macros into core + adapters | ||
time: 2022-09-26T12:36:09.319981-07:00 | ||
custom: | ||
Author: colin-rogers-dbt | ||
Issue: "483" | ||
PR: "480" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
kind: Fixes | ||
body: Password doesn't pass to server using LDAP connection via thrift (#310) | ||
time: 2022-09-26T11:28:57.306285-04:00 | ||
custom: | ||
Author: VShkaberda | ||
Issue: "310" | ||
PR: "396" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
changesDir: .changes | ||
unreleasedDir: unreleased | ||
headerPath: header.tpl.md | ||
versionHeaderPath: "" | ||
changelogPath: CHANGELOG.md | ||
versionExt: md | ||
versionFormat: '## dbt-spark {{.Version}} - {{.Time.Format "January 02, 2006"}}' | ||
kindFormat: '### {{.Kind}}' | ||
changeFormat: '- {{.Body}} ([#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-spark/issues/{{.Custom.Issue}}), [#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-spark/pull/{{.Custom.PR}}))' | ||
kinds: | ||
- label: Breaking Changes | ||
- label: Features | ||
- label: Fixes | ||
- label: Under the Hood | ||
- label: Dependencies | ||
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-spark/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-spark/pull/{{.Custom.PR}}))' | ||
- label: Security | ||
changeFormat: '- {{.Body}} ({{if ne .Custom.Issue ""}}[#{{.Custom.Issue}}](https://github.com/dbt-labs/dbt-spark/issues/{{.Custom.Issue}}), {{end}}[#{{.Custom.PR}}](https://github.com/dbt-labs/dbt-spark/pull/{{.Custom.PR}}))' | ||
custom: | ||
- key: Author | ||
label: GitHub Username(s) (separated by a single space if multiple) | ||
type: string | ||
minLength: 3 | ||
- key: Issue | ||
label: GitHub Issue Number | ||
type: int | ||
minLength: 4 | ||
- key: PR | ||
label: GitHub Pull Request Number | ||
type: int | ||
minLength: 4 | ||
footerFormat: | | ||
{{- $contributorDict := dict }} | ||
{{- /* any names added to this list should be all lowercase for later matching purposes */}} | ||
{{- $core_team := list "emmyoop" "nathaniel-may" "gshank" "leahwicz" "chenyulinx" "stu-k" "iknox-fa" "versusfacit" "mcknight-42" "jtcohen6" "dependabot[bot]" "snyk-bot" }} | ||
{{- range $change := .Changes }} | ||
{{- $authorList := splitList " " $change.Custom.Author }} | ||
{{- /* loop through all authors for a PR */}} | ||
{{- range $author := $authorList }} | ||
{{- $authorLower := lower $author }} | ||
{{- /* we only want to include non-core team contributors */}} | ||
{{- if not (has $authorLower $core_team)}} | ||
{{- $pr := $change.Custom.PR }} | ||
{{- /* check if this contributor has other PRs associated with them already */}} | ||
{{- if hasKey $contributorDict $author }} | ||
{{- $prList := get $contributorDict $author }} | ||
{{- $prList = append $prList $pr }} | ||
{{- $contributorDict := set $contributorDict $author $prList }} | ||
{{- else }} | ||
{{- $prList := list $change.Custom.PR }} | ||
{{- $contributorDict := set $contributorDict $author $prList }} | ||
{{- end }} | ||
{{- end}} | ||
{{- end}} | ||
{{- end }} | ||
{{- /* no indentation here for formatting so the final markdown doesn't have unneeded indentations */}} | ||
{{- if $contributorDict}} | ||
### Contributors | ||
{{- range $k,$v := $contributorDict }} | ||
- [@{{$k}}](https://github.com/{{$k}}) ({{ range $index, $element := $v }}{{if $index}}, {{end}}[#{{$element}}](https://github.com/dbt-labs/dbt-spark/pull/{{$element}}){{end}}) | ||
{{- end }} | ||
{{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -33,29 +33,12 @@ jobs: | |
DBT_INVOCATION_ENV: circle | ||
docker: | ||
- image: fishtownanalytics/test-container:10 | ||
- image: godatadriven/spark:2 | ||
- image: godatadriven/spark:3.1.1 | ||
environment: | ||
WAIT_FOR: localhost:5432 | ||
command: > | ||
--class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 | ||
--name Thrift JDBC/ODBC Server | ||
--conf spark.hadoop.javax.jdo.option.ConnectionURL=jdbc:postgresql://localhost/metastore | ||
--conf spark.hadoop.javax.jdo.option.ConnectionUserName=dbt | ||
--conf spark.hadoop.javax.jdo.option.ConnectionPassword=dbt | ||
--conf spark.hadoop.javax.jdo.option.ConnectionDriverName=org.postgresql.Driver | ||
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer | ||
--conf spark.jars.packages=org.apache.hudi:hudi-spark-bundle_2.11:0.9.0 | ||
--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension | ||
--conf spark.driver.userClassPathFirst=true | ||
--conf spark.hadoop.datanucleus.autoCreateTables=true | ||
--conf spark.hadoop.datanucleus.schema.autoCreateTables=true | ||
--conf spark.hadoop.datanucleus.fixedDatastore=false | ||
--conf spark.sql.hive.convertMetastoreParquet=false | ||
--hiveconf hoodie.datasource.hive_sync.use_jdbc=false | ||
--hiveconf hoodie.datasource.hive_sync.mode=hms | ||
--hiveconf datanucleus.schema.autoCreateAll=true | ||
--hiveconf hive.metastore.schema.verification=false | ||
- image: postgres:9.6.17-alpine | ||
environment: | ||
POSTGRES_USER: dbt | ||
|
@@ -80,6 +63,9 @@ jobs: | |
environment: | ||
DBT_INVOCATION_ENV: circle | ||
DBT_DATABRICKS_RETRY_ALL: True | ||
DBT_TEST_USER_1: "[email protected]" | ||
DBT_TEST_USER_2: "[email protected]" | ||
DBT_TEST_USER_3: "[email protected]" | ||
docker: | ||
- image: fishtownanalytics/test-container:10 | ||
steps: | ||
|
@@ -95,6 +81,9 @@ jobs: | |
environment: | ||
DBT_INVOCATION_ENV: circle | ||
ODBC_DRIVER: Simba # TODO: move env var to Docker image | ||
DBT_TEST_USER_1: "[email protected]" | ||
DBT_TEST_USER_2: "[email protected]" | ||
DBT_TEST_USER_3: "[email protected]" | ||
docker: | ||
# image based on `fishtownanalytics/test-container` w/ Simba ODBC Spark driver installed | ||
- image: 828731156495.dkr.ecr.us-east-1.amazonaws.com/dbt-spark-odbc-test-container:latest | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
[flake8] | ||
select = | ||
E | ||
W | ||
F | ||
ignore = | ||
W503 # makes Flake8 work like black | ||
W504 | ||
E203 # makes Flake8 work like black | ||
E741 | ||
E501 | ||
exclude = test |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
name: 🐞 Bug | ||
description: Report a bug or an issue you've found with dbt-spark | ||
title: "[Bug] <title>" | ||
labels: ["bug", "triage"] | ||
body: | ||
- type: markdown | ||
attributes: | ||
value: | | ||
Thanks for taking the time to fill out this bug report! | ||
- type: checkboxes | ||
attributes: | ||
label: Is this a new bug in dbt-spark? | ||
description: > | ||
In other words, is this an error, flaw, failure or fault in our software? | ||
If this is a bug that broke existing functionality that used to work, please open a regression issue. | ||
If this is a bug in the dbt-core logic, please open an issue in the dbt-core repository. | ||
If this is a bug experienced while using dbt Cloud, please report to [support](mailto:[email protected]). | ||
If this is a request for help or troubleshooting code in your own dbt project, please join our [dbt Community Slack](https://www.getdbt.com/community/join-the-community/) or open a [Discussion question](https://github.com/dbt-labs/docs.getdbt.com/discussions). | ||
Please search to see if an issue already exists for the bug you encountered. | ||
options: | ||
- label: I believe this is a new bug in dbt-spark | ||
required: true | ||
- label: I have searched the existing issues, and I could not find an existing issue for this bug | ||
required: true | ||
- type: textarea | ||
attributes: | ||
label: Current Behavior | ||
description: A concise description of what you're experiencing. | ||
validations: | ||
required: true | ||
- type: textarea | ||
attributes: | ||
label: Expected Behavior | ||
description: A concise description of what you expected to happen. | ||
validations: | ||
required: true | ||
- type: textarea | ||
attributes: | ||
label: Steps To Reproduce | ||
description: Steps to reproduce the behavior. | ||
placeholder: | | ||
1. In this environment... | ||
2. With this config... | ||
3. Run '...' | ||
4. See error... | ||
validations: | ||
required: true | ||
- type: textarea | ||
id: logs | ||
attributes: | ||
label: Relevant log output | ||
description: | | ||
If applicable, log output to help explain your problem. | ||
render: shell | ||
validations: | ||
required: false | ||
- type: textarea | ||
attributes: | ||
label: Environment | ||
description: | | ||
examples: | ||
- **OS**: Ubuntu 20.04 | ||
- **Python**: 3.9.12 (`python3 --version`) | ||
- **dbt-core**: 1.1.1 (`dbt --version`) | ||
- **dbt-spark**: 1.1.0 (`dbt --version`) | ||
value: | | ||
- OS: | ||
- Python: | ||
- dbt-core: | ||
- dbt-spark: | ||
render: markdown | ||
validations: | ||
required: false | ||
- type: textarea | ||
attributes: | ||
label: Additional Context | ||
description: | | ||
Links? References? Anything that will give us more context about the issue you are encountering! | ||
Tip: You can attach images or log files by clicking this area to highlight it and then dragging files in. | ||
validations: | ||
required: false |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
blank_issues_enabled: false | ||
contact_links: | ||
- name: Ask the community for help | ||
url: https://github.com/dbt-labs/docs.getdbt.com/discussions | ||
about: Need help troubleshooting? Check out our guide on how to ask | ||
- name: Contact dbt Cloud support | ||
url: mailto:[email protected] | ||
about: Are you using dbt Cloud? Contact our support team for help! | ||
- name: Participate in Discussions | ||
url: https://github.com/dbt-labs/dbt-spark/discussions | ||
about: Do you have a Big Idea for dbt-spark? Read open discussions, or start a new one | ||
- name: Create an issue for dbt-core | ||
url: https://github.com/dbt-labs/dbt-core/issues/new/choose | ||
about: Report a bug or request a feature for dbt-core |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
name: ✨ Feature | ||
description: Propose a straightforward extension of dbt-spark functionality | ||
title: "[Feature] <title>" | ||
labels: ["enhancement", "triage"] | ||
body: | ||
- type: markdown | ||
attributes: | ||
value: | | ||
Thanks for taking the time to fill out this feature request! | ||
- type: checkboxes | ||
attributes: | ||
label: Is this your first time submitting a feature request? | ||
description: > | ||
We want to make sure that features are distinct and discoverable, | ||
so that other members of the community can find them and offer their thoughts. | ||
Issues are the right place to request straightforward extensions of existing dbt-spark functionality. | ||
For "big ideas" about future capabilities of dbt-spark, we ask that you open a | ||
[discussion](https://github.com/dbt-labs/dbt-spark/discussions) in the "Ideas" category instead. | ||
options: | ||
- label: I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations) | ||
required: true | ||
- label: I have searched the existing issues, and I could not find an existing issue for this feature | ||
required: true | ||
- label: I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion | ||
required: true | ||
- type: textarea | ||
attributes: | ||
label: Describe the feature | ||
description: A clear and concise description of what you want to happen. | ||
validations: | ||
required: true | ||
- type: textarea | ||
attributes: | ||
label: Describe alternatives you've considered | ||
description: | | ||
A clear and concise description of any alternative solutions or features you've considered. | ||
validations: | ||
required: false | ||
- type: textarea | ||
attributes: | ||
label: Who will this benefit? | ||
description: | | ||
What kind of use case will this feature be useful for? Please be specific and provide examples, this will help us prioritize properly. | ||
validations: | ||
required: false | ||
- type: input | ||
attributes: | ||
label: Are you interested in contributing this feature? | ||
description: Let us know if you want to write some code, and how we can help. | ||
validations: | ||
required: false | ||
- type: textarea | ||
attributes: | ||
label: Anything else? | ||
description: | | ||
Links? References? Anything that will give us more context about the feature you are suggesting! | ||
validations: | ||
required: false |
Oops, something went wrong.