Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch package to use fsspec for backends #233

Draft
wants to merge 66 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
e7bbfa0
Add audbackend.backend.MinIO
hagenw Sep 3, 2024
cf34071
Add dependency to minio
hagenw Sep 3, 2024
dd642d0
Rename MinIO to Minio
hagenw Sep 4, 2024
40a2654
Fix Minio.delete() to first remove objects
hagenw Sep 4, 2024
64941c5
Let Minio.exists() return True for "/"
hagenw Sep 4, 2024
fe4b88d
Fix Minio.ls() for non-existing paths
hagenw Sep 4, 2024
054174e
Add tests using a local MinIO server
hagenw Sep 4, 2024
d19d390
Expand tests to 100% code coverage
hagenw Sep 4, 2024
07b34df
Fix remaining issues
hagenw Sep 4, 2024
3ed2685
Update docstrings
hagenw Sep 4, 2024
77b3a14
Define constant hosts values at single position
hagenw Sep 4, 2024
3a8517d
Run tests on play.min.io
hagenw Sep 4, 2024
7073080
Be more conservative for other copy method
hagenw Sep 4, 2024
c5e6a27
Add Minio.get_config()
hagenw Sep 4, 2024
393cff7
Add docstring example
hagenw Sep 4, 2024
2284cfc
Correct get_config() docstring
hagenw Sep 4, 2024
9faf954
Add secure argument
hagenw Sep 4, 2024
f16014e
Use fsspec as backend
hagenw Sep 5, 2024
aebdf90
Start working on fsspec backends
hagenw Sep 6, 2024
ab11f97
Complete first round of tests
hagenw Sep 9, 2024
01e0f02
Update interface base class
hagenw Sep 9, 2024
8672b6d
Continue with fsspec implementation
hagenw Sep 10, 2024
b4feac5
Add maven backend
hagenw Sep 10, 2024
e7c0951
Fix tests for unversioned backend
hagenw Sep 10, 2024
d88219d
Cleanup file
hagenw Sep 10, 2024
292faf6
Reformulate put/get_archive() in abstract class
hagenw Sep 10, 2024
42970e9
Fix tests for versioned backend
hagenw Sep 10, 2024
15e52a5
Add tests for Maven
hagenw Sep 10, 2024
fc1ebd8
Add extra test file for validation
hagenw Sep 10, 2024
47874bb
Remove unneeded test files
hagenw Sep 11, 2024
3c8f342
Fix validation tests
hagenw Sep 11, 2024
b93083a
Add test for broken filesystem
hagenw Sep 11, 2024
65906b3
Remove backend classes
hagenw Sep 11, 2024
f9df5d5
Remove interface/ folder
hagenw Sep 11, 2024
5e467e9
Add tests for NotImplementedError
hagenw Sep 11, 2024
8246ba2
Complete tests for Maven
hagenw Sep 11, 2024
ae2f399
Improve structure of validate tests
hagenw Sep 11, 2024
78f228b
Start working on doctsring tests
hagenw Sep 11, 2024
3075ae7
Use sybil for doctests
hagenw Sep 11, 2024
cbb714a
DOC: clean up API docs
hagenw Sep 11, 2024
11fc675
Started updating user docs
hagenw Sep 11, 2024
abe7528
Start updating usage docs
hagenw Sep 12, 2024
1975190
Fix usage documentation code
hagenw Sep 12, 2024
dd7c6c8
Update developer documentation
hagenw Sep 12, 2024
25cad0c
Cleanup docs
hagenw Sep 12, 2024
5807eeb
Fix doctests
hagenw Sep 13, 2024
dd7daa9
Remove filesystem only tets
hagenw Sep 13, 2024
f0469a9
Try to fix coverage
hagenw Sep 13, 2024
1d3363f
Try to fix coverage
hagenw Sep 13, 2024
12d1f19
Update coveragerc files syntax
hagenw Sep 13, 2024
9648b86
Update doctest for FileNotFoundError Windows
hagenw Sep 13, 2024
84d09b6
Use different file name in developer guide
hagenw Sep 13, 2024
911ff75
Try to fix shelve.open() on non Linux
hagenw Sep 13, 2024
8f62d59
Try to fix shelve for non Linux
hagenw Sep 13, 2024
878e5d4
Avoid using shelve
hagenw Sep 13, 2024
6260d2d
Replace Interface with Backend
hagenw Sep 13, 2024
0f5c2ca
Add option to switch fsspec in tests
hagenw Sep 13, 2024
ed9c90d
Define additional workflow for minio fs
hagenw Sep 13, 2024
eeb8575
Try to fix if statement in workflow
hagenw Sep 13, 2024
8490079
Try to fix skipping of minio test
hagenw Sep 13, 2024
265bfa0
Use Python 3.10 for minio test
hagenw Sep 13, 2024
15fa235
DEBUG
hagenw Sep 13, 2024
09fae50
DEBUG
hagenw Sep 13, 2024
1c77abc
Try to fix test
hagenw Sep 13, 2024
a2b0400
DEBUG
hagenw Sep 13, 2024
c15a569
DEBUG
hagenw Sep 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .coveragerc.Linux
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[report]
exclude_lines =
exclude_also =
pragma: no cover
pragma: no Linux cover
2 changes: 1 addition & 1 deletion .coveragerc.Windows
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[report]
exclude_lines =
exclude_also =
pragma: no cover
pragma: no Windows cover
2 changes: 1 addition & 1 deletion .coveragerc.macOS
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[report]
exclude_lines =
exclude_also =
pragma: no cover
pragma: no macOS cover
22 changes: 15 additions & 7 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ jobs:
python-version: '3.9'
- os: ubuntu-latest
python-version: '3.11'
- os: ubuntu-latest
python-version: '3.11'
fs: minio

steps:
- uses: actions/checkout@v4
Expand All @@ -35,29 +38,34 @@ jobs:
python -V
python -m pip install --upgrade pip

- name: Install packages with file-system backend
- name: Install dependencies
run: |
pip install -e .
pip install -r requirements.txt
pip install -r tests/requirements.txt

- name: Test file-system only package
- name: Default file system
run: |
python -m pytest --cov-fail-under=0 tests/test_backend_filesystem_only.py tests/test_backend_filesystem.py
echo "AUDBACKEND_TEST_FS=dir" >> $GITHUB_ENV

- name: Install package with all dependencies
- name: Use minio file system
run: |
pip install -r requirements.txt
ping -c1 -W1 play.minio.io >/dev/null && FS="minio" || FS="none"
echo "AUDBACKEND_TEST_FS=${FS}" >> $GITHUB_ENV
if: matrix.fs == 'minio'

- name: Test with pytest
env:
ARTIFACTORY_USERNAME: ${{ secrets.ARTIFACTORY_USERNAME }}
ARTIFACTORY_API_KEY: ${{ secrets.ARTIFACTORY_API_KEY }}
run: |
# export AUDBACKEND_TEST_FS=${{ env.AUDBACKEND_TEST_FS }}
echo $AUDBACKEND_TEST_FS
python -m pytest --cov-config=.coveragerc.${{ runner.os }}
if: ${{ env.AUDBACKEND_TEST_FS != 'none' }}

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./coverage.xml
if: matrix.os == 'ubuntu-latest'
if: ${{ (matrix.os == 'ubuntu-latest') && (env.AUDBACKEND_TEST_FS != 'none') }}
19 changes: 4 additions & 15 deletions audbackend/__init__.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,8 @@
from audbackend import backend
from audbackend import interface
from audbackend.core.api import access
from audbackend.core.api import create
from audbackend.core.api import delete
from audbackend.core.api import register
from audbackend.core.backend.base import Base as Backend # legacy
from audbackend.core.backend.filesystem import FileSystem # legacy
from audbackend.core.base import AbstractBackend
from audbackend.core.errors import BackendError
from audbackend.core.repository import Repository

# Import optional backends (legacy)
try:
from audbackend.core.backend.artifactory import Artifactory
except ImportError: # pragma: no cover
pass
from audbackend.core.maven import Maven
from audbackend.core.unversioned import Unversioned
from audbackend.core.versioned import Versioned


__all__ = []
Expand Down
8 changes: 0 additions & 8 deletions audbackend/backend/__init__.py

This file was deleted.

235 changes: 0 additions & 235 deletions audbackend/core/api.py

This file was deleted.

Empty file.
Loading