Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[FEAT] ls/list_dir for AzureBlobStorage (#1408)
File listing for AzureBlobStorage via list_blobs(prefix) API. Closes - #1439 Manually checked: - scheme:// -> all buckets - scheme://bucket -> ls the bucket - scheme://bucket/ -> ls the bucket - scheme://bucket/dir -> ls the dir - scheme://bucket/dir/ -> ls the dir - scheme://bucket/dir// -> ls the dir - scheme://bucket/file.txt -> returns a single File entry - scheme://bucket/emptydir -> returns [] - scheme://bucket/MISSING should throw FileNotFoundError - buckets and folders are returned with trailing slashes Implementation notes: - This reimplements iter_dir of the ObjectSource trait, since the Azure Rust SDK exposes paginated streams directly (instead of continuation tokens). - Supports both protocols that fsspec supports (`az://` and `abfs://`). Whichever protocol used is persisted. - Supports both URI formats that fsspec supports (`protocol://container/path-part/file` and `protocol://[email protected]/path-part/file`), but all results are translated to the first format. --------- Co-authored-by: Xiayue Charles Lin <[email protected]>
- Loading branch information