Clarifying store list_dir / listdir details in implementation vs. spec

[rabernat]

Here's what the V3 Spec currently says about the store list_dir method

``list_dir`` - Retrieve all keys and prefixes with a given prefix and
which do not contain the character "/" after the given prefix.

    | Parameters: `prefix`
    | Output: set of `keys` and set of `prefixes`

    For example, if a store contains the keys "a/b", "a/c", "a/d/e",
    "a/f/g", then ``list_dir("a/")`` would return keys "a/b" and "a/c"
    and prefixes "a/d/" and "a/f/". ``list_dir("b/")`` would return
    the empty set.

https://github.com/alimanfoo/zarr-specs/blob/b63e64dcf530f195b19691e06a44c85916f33291/docs/core/v3.0.rst?plain=1#L1344-L1353

There are three inconsistencies between this and the current zarr-python V3 implementation:

• The function is named listdir (not list_dir) in the implementation
• The listdir function does not include the prefix. So listdir("a/") returns b, not a/b. This is pretty important for custom store implementations.
• The implementation do not return a trailing slash for prefixes, while the spec seems to recommend this.

I think we should make the spec and the implementation consistent.

cc @jstriebel and @alimanfoo, who are currently revising the V3 spec.

[jstriebel]

@rabernat Do you have any preferences, e.g. should the python implementation change or rather the spec? I don't have a strong opinion which version is more appropriate, but agree that using this coherently would be nice.

[jstriebel]

After re-reading this part of the spec, I think that the / suffix is useful in the API, especially to make this definition as simple as it is:

Retrieve all keys and prefixes with a given prefix and
which do not contain the character "/" after the given prefix.

@rabernat @joshmoore Should we turn this into a zarr-python issue then? (Related to zarr-developers/zarr-python#1290)

[rabernat]

I agree it is a python issue.

I'm still unsure of how to distinguish keys vs. prefixes and whether this needs to be specified in the spec. This is related to #200. Ideally we should suggest a specific algorithm for efficiently traversing a store using listdir. Right now the zarr-python implementation makes tons of unnecessary list and __contains__ calls.

[joshmoore]

I'm still unsure of how to distinguish keys vs. prefixes and whether this needs to be specified in the spec.

👍 This would be good to capture and discuss for v3.

[jstriebel]

@rabernat @joshmoore Do you have any concrete proposals for the current spec? Happy to do a review, but I'm not quite sure what exactly should be changed.

[rabernat]

Basically the store needs a way to communicate whether the thing it is returning is a file or a directory. One way to solve this would to have prefixes / directories always return a trailing / in a list operation. But that feels like an implementation detail, rather than a spec issue.