Skip to content

Commit

Permalink
[FEAT] ls/list_dir for AzureBlobStorage (#1408)
Browse files Browse the repository at this point in the history
File listing for AzureBlobStorage via list_blobs(prefix) API. Closes 
- #1439 

Manually checked:

- scheme:// -> all buckets
- scheme://bucket -> ls the bucket
- scheme://bucket/ -> ls the bucket
- scheme://bucket/dir -> ls the dir
- scheme://bucket/dir/ -> ls the dir
- scheme://bucket/dir// -> ls the dir
- scheme://bucket/file.txt -> returns a single File entry
- scheme://bucket/emptydir -> returns []
- scheme://bucket/MISSING should throw FileNotFoundError
- buckets and folders are returned with trailing slashes

Implementation notes:
- This reimplements iter_dir of the ObjectSource trait, since the Azure
Rust SDK exposes paginated streams directly (instead of continuation
tokens).
- Supports both protocols that fsspec supports (`az://` and `abfs://`).
Whichever protocol used is persisted.
- Supports both URI formats that fsspec supports
(`protocol://container/path-part/file` and
`protocol://[email protected]/path-part/file`), but
all results are translated to the first format.

---------

Co-authored-by: Xiayue Charles Lin <[email protected]>
  • Loading branch information
xcharleslin and Xiayue Charles Lin authored Sep 29, 2023
1 parent 069432d commit 831d45c
Showing 1 changed file with 344 additions and 28 deletions.
Loading

0 comments on commit 831d45c

Please sign in to comment.