Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support git filesystem in filesystem source #301

Open
rudolfix opened this issue Dec 12, 2023 · 1 comment · May be fixed by dlt-hub/dlt#954
Open

support git filesystem in filesystem source #301

rudolfix opened this issue Dec 12, 2023 · 1 comment · May be fixed by dlt-hub/dlt#954
Assignees
Labels
source new feature adds new feature to existing source

Comments

@rudolfix
Copy link
Contributor

Which source is this for
adds git filesystem support to filesystem source

Describe the data you'd like, be specific, link the docs
We want to support git in filesystem source the same way we support buckets. This will involve using (and possibly reimplementing) the filesystem provided by fsspec by default.

    • try if the default git filesystem implementation works, if not we'll need to implement our own. the problem I see is that last modification timestamp of the file in git repo is not returned by the default fsspec impl.
    • initially you can add all the code (including custom fsspec filesystem) to the filesystem verified source
    • if you need to modify library core, do it a fork from which we can install it for testing. after a review we'll merge it
    • please reproduce all the test cases by creating a repo with the test files used by other filesystems in filesystem source
@deanja
Copy link
Contributor

deanja commented Dec 20, 2023

@rudolfix I am trying GitPython as the basis for a custom fsspec implementation. You mentioned that dlt already uses it (see point 2 above) . Just want to make sure because some GitPython methods require the user to have git binaries installed.

ok to use GitPython - WDYT?

deanja added a commit to deanja/dlt-verified-sources that referenced this issue Dec 26, 2023
@sh-rp sh-rp moved this from Planned to Implementation & Review in Verified Sources Jan 8, 2024
@rudolfix rudolfix moved this from Implementation & Review to Ready for Deployment in Verified Sources Mar 18, 2024
@rudolfix rudolfix moved this from Ready for Deployment to Implementation & Review in Verified Sources Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
source new feature adds new feature to existing source
Projects
Status: Implementation & Review
Development

Successfully merging a pull request may close this issue.

2 participants