23rd February, 2022

Attending: Dennis, Josh, Sanket, Jeremy, Ward, Hailey Johnson, John K., Eric Perlman, Greg Lee

  • Updates from Sanket

  • Release of numcodecs incl.

    • Entrypoints (if Martin shows up)

    • 307

      flatten: need an attribute?

      • John: Unsure. zfpy remembers internally.
    • 305

      ndarray-like

    • HJ: NetCDF meeting on codecs soon. Good to know prioritization

      • blosc of course
    • JM: Twitter poll? “What compressor do you frequently use in

      Zarr?”

    • EP: discussed with d-v-b that lossy compressors would be nice.

      • jpeg chunk storage
      • JK: tried zfpy? No. from seung lab. (Meteorological data)
    • JMS: imagecodecs from Gohlke.

      more explicit std. in v3 (need json for each parameter)

  • Jeremy (in order of importance)

    • 1. [*consistency in referring to coordinates / dimension

      order*](https://github.com/zarr-developers/zarr-specs/issues/129)

      • a. Unambiguously referring to dimensions / coordinates (top)

        • Ward: in NC order of dimensions is under the hood,

          indexing into set of arrays independently of the underlying order. file written by netcdf-fortran should be indistinguishable from by netcdf-c (due to work in the library to be machine/language-independent)

        • JMS: tension cf. Julia’s desire of order

        • WF: similar issue with endianness? JMS: big endian

          likely dead WF: sadly no. see netcdf repo for redhat

      • b. Support for different storage orders: https://github.com/zarr-developers/zarr-specs/issues/129

      • c. Support for non-zero origin

    • 2. appetite for zarr multiscale spec?

      • Josh: xarray/datatree status
      • Move metadata from .zattrs to .zarray?
      • JMS: primarily getting it outside of OME, in discrete state

    • 3. Data type syntax

    • 4. URL syntax

  • notes in https://hackmd.io?

    • Yes: Josh, Greg, Ward, …

    • No:

    • John: not all in one document

    • “Motion carries”

  • Dennis: could use help on how to work struct into extension

    • https://github.com/zarr-developers/zarr-specs/issues/49#issuecomment-1047270856

    • JMS: one of the big changes in V3 is no support for structs

      (numpy array)

    • GL: several datatypes are no in v3. currently missing a document

      describing those types. e.g. https://github.com/zarr-developers/zarr-python/pull/898

    • JMS: will numpy be supported in V3?

      • GL: was easy to implement but they aren’t written up as specs yet
      • Untested are: unicode and bytestring (only indirectly in xarray)

    • JMS: numpy doesn’t support variable length strings

    • JMS: numpy struct datatypes lead to interleaved in memory. not

      great for compression. perhaps better to transpose them. would be nice to not be tied to the numpy model.

    • JK: need something in the spec, which is why it was left out of

      the spec so far.

    • GL: just a few types currently in v3. complex would be easy to

      support. can choose what goes in or not. (or warn or error …)

    • DH: trying to figure out how to support as much of NC/HDF5 core

      data model as possible. Big missing piece are: structs, enumerations, vlenstrings, vlenobjects (sequences)

    • JMS: as multiple arrays? DH: question of how to specify in the

      extension mechanism. (lots of possible implementations)

  • Feedback process (!)

    • https://github.com/zarr-developers/governance/issues/14

    • Sanket: spoke to Alistair

    • Suggestions welcome.

  • (Tabled)

    • Fill_value issues

    • [*change in indexing

      behavior*](https://github.com/zarr-developers/zarr-python/issues/967)