NCZarr Current and Future Extensions to pure Zarr.

[DennisHeimbigner]

Current Extensions:

• shared named dimensions
• character type
• typed attributes
  • allows an attribute to have an arbitrary type (as opposed to the implicit
    string type of Zarr).
• Scalar variables
• Superblock to record properties about a dataset.
• Each group keeps a record of its contained objects: dimensions, arrays (aka variables) and sub-groups.

Future Extensions:

• Unlimited dimensions
  • this is a dimension whose length can be increased dynamically.
  • if the dimension is increased then it applies to every array
    that uses that dimension.
• User defined shared types
  • Enumerations
  • Structs
  • Variable length vectors of objects
  • Variable length strings

[jbms]

This summary is helpful --- thanks. Can you clarify a few details, though? See inline comments below.

Current Extensions:

• shared named dimensions
• character type

NumPy (and therefore zarr-python / zarr v2) has some character types, but can you say a bit more how this works / how it differs from what is built into NumPy?

• typed attributes

  • allows an attribute to have an arbitrary type (as opposed to the implicit
    string type of Zarr).

• Scalar variables

Do you mean an array with zero dimensions (rank 0)? Zarr v2 already supports this.

• Superblock to record properties about a dataset.
• Each group keeps a record of its contained objects: dimensions, arrays (aka variables) and sub-groups.

Future Extensions:

• Unlimited dimensions

  • this is a dimension whose length can be increased dynamically.
  • if the dimension is increased then it applies to every array
    that uses that dimension.

In zarr v2 all dimensions are already resizeable, but of course there is no sharing of dimensions between multiple arrays.

• User defined shared types

Can you explain what a shared type would be?

• Enumerations
• Structs
• Variable length vectors of objects
• Variable length strings

[DennisHeimbigner]

Are we looking at the same spec?
I am using https://zarr.readthedocs.io/en/stable/spec/v2.html
and as near as I can tell is says nothing about resizing dimensions (I assume you mean
changing the shape of an array), nor about rank 0 arrays.

[jbms]

Are we looking at the same spec? I am using https://zarr.readthedocs.io/en/stable/spec/v2.html and as near as I can tell is says nothing about resizing dimensions (I assume you mean changing the shape of an array), nor about rank 0 arrays.

It is true that the spec is rather limited --- I have relied on the zarr-python source code for clarification when implementing zarr.

For rank 0 arrays, you can simply specify "shape": [], "chunks": []. The spec does not mention it, but zarr-python names the chunk "0" as a special case (since naming it "" would be problematic for some stores).

Resizing is not specifically mentioned in the spec, but given that chunks are always full size, you can simply change the metadata to have a new size, and delete existing out-of-bounds chunks if you care to do so.

[jstriebel]

#188 is now open to improve the docs regarding resizing

[DennisHeimbigner]

Also, here I see no character type
https://numpy.org/doc/stable/reference/arrays.interface.html#arrays-interface
To what are you referring?

[jbms]

Also, here I see no character type https://numpy.org/doc/stable/reference/arrays.interface.html#arrays-interface To what are you referring?

Sorry, I had seen NPY_CHAR (type code "c", which is particularly weird since "c8" and "c16" are complex number types) but that was deprecated a long time ago --- now there is just "|S1", fixed-length string of length 1, and "|U1", fixed-length unicode string of 1 code point, dtype("c") returns dtype("S1"). Since Python itself has no separate types for characters as opposed to length-1 strings, eliminating the distinction in NumPy also doesn't really affect anything, but in other languages it may indeed be important to distinguish between char type and length 1 string.

[joshmoore]

@DennisHeimbigner: also a 👍 from my side for the list. Two questions that could may be reflected in the original post:

• is there a link for each of these that you could recommend?
• do you have a breakdown of which you would see proposing as Zarr extensions?

[gdkrmr]

Typed attributes are necessary for proper support of the CF convetions, see cf-convention/cf-conventions#422.

[briannapagan]

@DennisHeimbigner I have also sent you an email, but this would be great to take up within the GeoZarr spec working group. https://github.com/zarr-developers/geozarr-spec