[Feature] On Linux: respect XDG_CONFIG_HOME and don't store config in user's home directory

[sethwoodworth]

Describe the feature

The XDG Base Directory standard states that user config files should respect the XDG_CONFIG_HOME environment variable, which defaults $HOME/.config.

Currently, dbt defaults the PROFILE_DIR to ~/.dbt and doesn't attempt to read from XDG_CONFIG_HOME or ~/.config.

mv ~/.dbt ~/.config/dbt
dbt debug
...
profiles.yml file [ERROR not found]

Describe alternatives you've considered

I can locally symlink my ~/.config/dbt directory to ~/.dbt, but it's not my preference to put additional directories in my home directory. Alternately, I could set a global DBT_PROFILES_DIR variable.

Additional context

This is only relevant to Linux users. The appdirs pypi package makes the XDG_Base_directory standard simple to implement.

Who will this benefit?

Linux users who have far too many files/folders in their home directories.

[drewbanin]

Thanks for opening this issue @sethwoodworth. Looping in @beckjake to take a look :)

[beckjake]

I'm surprised we don't have an issue about this yet, thanks for writing it up!

I agree, also Windows should write to %APPDATA%\dbt or something like that, and macos should write to ~/Library/Application Support/dbt (I think?). It looks like that library you linked might help on that front.

The problem here is going to be migration. This fix is going to break stuff for a lot of people! We should do it before 1.0. We can fall back to ~/.dbt/profiles.yml if we don't find it in the os-specific directory. I assume we'll still get a lot of complaints this way, but better now than later.

[sethwoodworth]

I was unaware of the DBT_PROFILES_DIR environment variable. A simple workaround on linux is to set the variable with:

export DBT_PROFILES_DIR="$(XDG_CONFIG_DIR)/dbt"

in your .zprofile or etc

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[dradetsky]

I found this trying to figure out why dbt didn't support xdg properly. It really should.

[dradetsky]

@dbeatty10 you closed the issue as "not planned" but didn't touch my PR (#11108). Did you mean to reject it? If so, was this like a "Yeah, that'd be nice but we don't have time to do this and make sure it's 100% correct just now" kinda thing? Or was there some reason why y'all didn't like it?

[dbeatty10]

Hey @dradetsky !

Quick explanation: the stale bot closed this as "completed" (done, closed, fixed, resolved) on Jun 23, 2022, but it should have closed it as "not planned" (won't fix, can't repro, duplicate, stale):

(This affects a lot of older issues that were closed by the stale bot, but it is does not affect more recent issues closed by the stale bot.)

So I made a quick fix to reflect that it was not completed, but I probably should have also then immediately re-opened for further triage due to your linked PR. Re-opening now 👍

[dbeatty10]

@dradetsky I removed the good_first_issue label as we further assess if this is something we'd be still interested in adopting or not.

Overview

The folder location where dbt searches for connection profiles is absolutely fundamental to how dbt operates, so any changes in this area of code introduce substantial risk of regressions. So it isn't a change we can take lightly.

Workaround

Did you consider or try out the suggestion in #2515 (comment)?

export DBT_PROFILES_DIR="$(XDG_CONFIG_HOME)/dbt"

This looks like a pretty lightweight approach to look in the conventional XDG folder without needing to make any changes to dbt itself.

Unless overridden by the --profiles CLI flag, dbt will look in the folder specified by DBT_PROFILES_DIR for database connection profiles [1].

[dradetsky]

The folder location where dbt searches for connection profiles is absolutely fundamental to how dbt operates, so any changes in this area of code introduce substantial risk of regressions

Yes, a proper fix for this would probably involve automatic migration of legacy profile dirs, or falling back to the legacy dir if the primary dir doesn't have the profiles, or both. I didn't really want to write all that without talking to someone given that this was likely going to involve talking to DX peoples anyhow.

Did you consider or try out the suggestion

I mean, no, but I saw it and I'm sure it would work if all I wanted was to fix this for me asap. I just did this on the grounds that using xdg dirs is the Correct Thing(tm) and dbt should do the Correct Thing(tm) if possible. I also thought (but didn't know for sure) that the defaults used by appdirs for win/mac would also be an improvement over the status quo.

Using the env var is probably what you should recommend in the short term. I would still recommend that the change in my PR (along with necessary migration changes) should be made. However, that will involve getting some stakeholders to agree on what kinds of migration/fallback/testing is necessary to get P(regression) down to whatever value said stakeholders think is good enough, so it can't be done immediately.

For that reason, I think I'll pull out the change that fixes the circular import issue & induces copypasta and submit that. That will make any 2nd PR smaller & simpler, which is always good.