From 40461c205203f486e90d9b3dd31815ea4394a786 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Fri, 23 Aug 2024 10:33:54 -0500 Subject: [PATCH 1/4] Added consolidated metadata to spec --- docs/v3/core/v3.0.rst | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/v3/core/v3.0.rst b/docs/v3/core/v3.0.rst index 4f8d256..a34707f 100644 --- a/docs/v3/core/v3.0.rst +++ b/docs/v3/core/v3.0.rst @@ -806,6 +806,40 @@ Optional keys: pairs, where the key must be a string and the value can be an arbitrary JSON literal. Intended to allow storage of arbitrary user metadata. + +``consolidated_metadata`` +^^^^^^^^^^^^^^^^^^^^^^^^^ + + An object consolidating all the Array and Group metadata of members below + the root node in a hierarchy. + + =============== ========================= =================================================================================================== + Field Type Description + =============== ========================= =================================================================================================== + metadata ``Map`` A mapping from node path to Group or Array ``Metadata`` object. + kind const ``'inline'`` The string literal ``'inline'``. Reserved for future use. + must_understand const ``False`` The boolean literal ``False``. Indicates that the field is not required to load the Zarr hierarchy. + =============== ========================= =================================================================================================== + + Consolidated metadata can help reduce the time needed to load the metadata + for an entire hierarchy, especially when the metadata is being served over a + network. Without consolidated metadata, opening an entire hierarchy over the + network requires an HTTP request per node. Consolidated metadata enables + loading the metadata for every node in a hierarchy with a single HTTP + request. + + Consolidated Metadata is optional. If present, then readers should use the + consolidated metadata when . When not present, readers should use the + non-consolidated metadata located in the Store to load the data. + + The ``kind`` field indicates that consolidated metadata is stored inline in + the root ``zarr.json`` object. At this time, ``'inline'`` is the only + supported value for ``kind``. Future versions of the specification may allow + for consolidated metadata in other locations. + +Example Group Metadata +---------------------- + For example, the JSON document below defines an explicit group:: { From 4dfce82cdb2ac3bd65e4fee9922b728e43280ca9 Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Mon, 26 Aug 2024 09:49:24 -0500 Subject: [PATCH 2/4] fixup --- docs/v3/core/v3.0.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/v3/core/v3.0.rst b/docs/v3/core/v3.0.rst index a34707f..de6863c 100644 --- a/docs/v3/core/v3.0.rst +++ b/docs/v3/core/v3.0.rst @@ -821,6 +821,10 @@ Optional keys: must_understand const ``False`` The boolean literal ``False``. Indicates that the field is not required to load the Zarr hierarchy. =============== ========================= =================================================================================================== + Note that *all* children Arrays and Groups should be included in + consolidated metadata, not just the nodes immediately below the root Group. + Children nested inside other groups should be included too. + Consolidated metadata can help reduce the time needed to load the metadata for an entire hierarchy, especially when the metadata is being served over a network. Without consolidated metadata, opening an entire hierarchy over the From 230e8cea204e00149e8223de5b2f487003e2bb2d Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Wed, 2 Oct 2024 10:56:08 -0500 Subject: [PATCH 3/4] Clarify node keys --- docs/v3/core/v3.0.rst | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/v3/core/v3.0.rst b/docs/v3/core/v3.0.rst index 7bd0b56..3c99876 100644 --- a/docs/v3/core/v3.0.rst +++ b/docs/v3/core/v3.0.rst @@ -817,10 +817,6 @@ Optional keys: must_understand const ``False`` The boolean literal ``False``. Indicates that the field is not required to load the Zarr hierarchy. =============== ========================= =================================================================================================== - Note that *all* children Arrays and Groups should be included in - consolidated metadata, not just the nodes immediately below the root Group. - Children nested inside other groups should be included too. - Consolidated metadata can help reduce the time needed to load the metadata for an entire hierarchy, especially when the metadata is being served over a network. Without consolidated metadata, opening an entire hierarchy over the @@ -828,6 +824,30 @@ Optional keys: loading the metadata for every node in a hierarchy with a single HTTP request. + Note that *all* children Arrays and Groups should be included in + consolidated metadata, not just the nodes immediately below the root Group. + Children nested inside other groups should be included too as a flat list of + nodes. The keys of ``metadata`` should be the path of the node relative to + the node at which the metadata is being consolidated (i.e. the ``Group`` + where this ``consolidated_metadata`` object is stored). For example, given a + hierarchy with groups like the following, where capital letters indicate + groups: and lowercase letters indicate arrays:: + + A/ + B/ + C/ + x + y + + If we consolidate the metadata at the Group ``A``, the consolidated metadata + would have the keys ``"A", "A/B", "A/B/C", "A/B/C/x", ...``. + + If we consolidate the metadata at the Group ``B``, the consolidated metadata + would have the keys ``"C", "C/x", "C/y"``. + + If we consolidate the metadata at the Group ``C``, the consolidated metadata + would have the keys ``"x", "y"``. + Consolidated Metadata is optional. If present, then readers should use the consolidated metadata when . When not present, readers should use the non-consolidated metadata located in the Store to load the data. From db269ae42e20cc5ce531118716d498c367b9216c Mon Sep 17 00:00:00 2001 From: Tom Augspurger Date: Thu, 3 Oct 2024 12:28:49 -0500 Subject: [PATCH 4/4] fixed typo --- docs/v3/core/v3.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/v3/core/v3.0.rst b/docs/v3/core/v3.0.rst index 3c99876..0ff9124 100644 --- a/docs/v3/core/v3.0.rst +++ b/docs/v3/core/v3.0.rst @@ -849,7 +849,7 @@ Optional keys: would have the keys ``"x", "y"``. Consolidated Metadata is optional. If present, then readers should use the - consolidated metadata when . When not present, readers should use the + consolidated metadata. When not present, readers should use the non-consolidated metadata located in the Store to load the data. The ``kind`` field indicates that consolidated metadata is stored inline in