Add iter_keys_upto_delimiter #93
Conversation
|
@mbr : This seems like a useful addition to me and complements the prefix support in |
|
I'm not entirely sure this needs to be in the core API (I like to keep it small, as you know ;)). The "primitive" implementation is just iterating over all keys and collecting the prefix. While this is useful, is there a real need to add this to the core API? There is one condition under which I would add it: If it is common for backend-implementations to offer a more efficient version of this (I think S3 might do this, but they might limit it to a fixed |
|
@mbr I've added fast-paths for Azure (all possible delimiters) and FS (only works if the delimiter is the OS path separator). For S3, this also works for any delimiter (see boto3 docs and AWS API docs), but I did not implement this since I think we should tackle #84 first and I'm not an S3 user. So yes, it is rather common for upstream services to support this "directory-like" listing. I did some internal benchmarks for massive stores and you get huge advantages from the fast-path:
|
|
Nice. In that case you have my blessings, I shall bikeshed no more =). Probably could not hurt to mention these things somewhere in the docs, i.e. that some backends might have better performance characteristics than others for this. |
simplekv/__init__.py
Outdated
| @@ -108,6 +108,33 @@ def iter_keys(self, prefix=u""): | |||
| """ | |||
| raise NotImplementedError | |||
|
|
|||
| def iter_prefixes_upto_delimiter(self, delimiter, prefix=u""): | |||
There was a problem hiding this comment.
Would iter_prefixes work here as well? If so, shorter is better.
simplekv/decorator.py
Outdated
| @@ -58,6 +58,20 @@ def iter_keys(self, prefix=u""): | |||
| return (self._unmap_key(k) for k in self._dstore.iter_keys(self._map_key_prefix(prefix)) | |||
| if self._filter(k)) | |||
|
|
|||
| def iter_prefixes_upto_delimiter(self, delimiter, prefix=u""): | |||
|
done |
PR is split in the following way:
Closes #83