[CT-2393] [Bug] On MacOS, running dbt functional tests fails with errors opening files

[ezraerb]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I downloaded a new clone of the dbt-core repository. It has no edits. Running 'make integration' runs approximately 17% of the tests, and then all tests fail. The error logs contain a mixture of seemingly unrelated errors: warnings can not be written to the log file, the sqllte file used by coverage can not be opened, and others. Lots of trial and error finally revealed that the running the tests is exceeding the open file descriptor (soft) limit. On MacOS, this limit is only 256.

Expected Behavior

Either the integration tests run without error on a newly created clone of the repository, or CONTRIBUTING.md has a note in the testing section about the need for a high open file descriptor limit.

Steps To Reproduce

1. On MacOS Monterrey, install development tools as specified in CONTRIBUTING.md
2. Clone the dbt-core repo
3. 'make integration'. See all the seemingly random file opening errors

Unlike step 3 above, 'pytest tests/functional' produces a clear error message that the file descriptor limit has been hit.

Relevant log output

No response

Environment

- OS: MacOS Monterrey 12.6.5
- Python: 3.9.6
- dbt: 1.5.0-b5

Which database adapter are you using with dbt?

postgres

Additional Context

No response

[dbeatty10]

Thanks for reporting this @ezraerb !

Trying out these instructions with macOS 12.6 worked fine for me the first time I tried it. Here are the limits my system reported:

$ ulimit -n               
1048575

$ launchctl limit maxfiles
        maxfiles    256            unlimited      

But after adjusting the file descriptor limit down by running ulimit -n 256 per these instructions, I was able to reproduce what you reported.

If we were to add a note in the testing section about the need for a high open file descriptor limit, do you have any suggestions for what that note would say?

I'm going to re-label this as an enhancement, because we reserve bug for when dbt-core itself is not working as expected.

[ezraerb]

I'm a little concerned about why the tests need so many open file descriptors, since they (hopefully) don't actually open that many resources at once. Are descriptors leaking somewhere? Since the issue has such an obvious work-around, I suspect the full fix to be a very low priority.

The method to raise the file descriptor limit varies by OS, so I would go with a generic description of the issue and its workaround: "Running functional tests can fail with repeated errors opening files and other resources. If this occurs, check directory permissions and the limit on open file descriptors. The latter needs to be at least 8000."

If the wording is acceptable, I'm happy to submit a PR with the change.

[dbeatty10]

That language would work for a PR 👍

Our integration testing framework uses pytest to create and run stand-alone dbt projects. There's probably hundreds of these.

One way to see more useful error messages:

ulimit -n 256
python3 -m pytest tests/functional

[ezraerb]

I'll submit the PR to document the workaround, and leave this ticket open for the fuller investigation.

[ezraerb]

PR submitted for the workaround note: #7343

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[dbeatty10]

Here's the workaround within dbt-duckdb within conftest.py:

soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (hard_limit, hard_limit))