2nd June, 2021

Attendees: Josh, Ward, Jackson, Hailey, Dennis, John

• Introductions

• Community items?

  • Papers & benchmarks (aicsimageio!)

  • OME meeting: “did you think of zarr”

  • Messaging (and all the papers). EOSS money?

  • Conferences in the Winter season: AGU,

    EGU, …

    • Talking about NetCDF & Zarr. Good chance to coordinate.
    • Very large, 25K attendees! “Earth science” incl. a lifescience component
    • Ward to bring it back to our attention when proposals are open.
    • “Earth and space science informatics” section. Daily posters.
    • “Technologies as they apply to a domain” (10+5)
    • Also cross-discipline sessions.

• Core

  • Josh (Py): Dependabot & Python 3.6

    • John: dropped in xarray, dask, tifffile, numpy
    • Josh to open an issue
    • https://github.com/zarr-developers/governance/issues/11
    • https://github.com/zarr-developers/community/issues

  • Josh (Py): Review backlog

    • e.g. Azure
    • “Mutable mapping from all cloud providers” ❤️
    • Use of owners?
    • https://github.blog/2017-07-06-introducing-code-owners/
    • via blame … separate files.
    • https://github.com/zarr-developers/zarr-python/pull/759 → storage into a package and then separate files.
    • Start with an issue? → https://github.com/zarr-developers/zarr-python/issues/764

  • Dennis (C)

    • How proposing to handle numcodec in nczarr. https://github.com/zarr-developers/zarr-specs/issues/41#issue comment-848222877

    • Also how to store structs? https://github.com/zarr-developers/zarr-specs/issues/118

      • https://github.com/saalfeldlab/n5/pull/73#issuecomment-688731487

      • Dennis: in HDF5 you can specify the padding. In python

        is all tightly packed? (Would prefer to stick with tightly packed)

      • John:

        https://numpy.org/doc/stable/user/basics.rec.html#automatic-byte-offsets-and-alignment is the definition. Possibly configurable. Would need input to know what is important.

    • Primarily working on filters for netcdf

    • Exposing numcodec API through netcdf (i.e. specifying filter through JSON). Not supported at the moment. Need to use HDF5 mechanism.

      • John: support for categories (e.g. enums) that e an ID.

      • Dennis: use characters (enums) for it.

      • John: testing of various implementations in

        https://github.com/zarr-developers/zarr_implementations/tree/master/generate_data

    • Critical to get users is the interoperability with HDF5. A bit stuck.

    • Even more important is xarray/zarr format. (More people are using it)

    • Need maximal interoperability with V2. V3 is on the horizon. (Won’t be there for quite a while)

    • True for Java as well. Will be reading V2 for quite a while.

  • Hailey (Java)

    • Agreed. Need v2.
    • Josh: update the spec & get zarr_implementations
    • Dennis: live without structs. filters is important.
    • Josh: so hearing that numcodecs registry is most important? https://github.com/zarr-developers/numcodecs/issues/278
    • Dennis: most critical piece is officially change numcodec representation to optionally include HDF5 ID. → alias field.

• Misc

  • Josh: h5coro (for Martin) - tabled