5th May, 2021

zarrR (community#18) TABLED

Attendees: Josh, John K., Ryan W., Dominik Lindner, Jean-marie Burel, Stephan Saalfeld, Tobi, Jean-Karim Heriche, Hailey Johnson, Dennis, Martin Durant, Greg Lee, Jackson Brown

  • Introductions

    • OME team: interest in workflows & analysis

    • J-K (EMBL): big R user, doing imaging

    • Tobi: looking for how to run Zarr on ipfs

  • R implementations

    • …links here…

  • Anything else?

    • nczarr & compressor support ([*4

      slides*](https://docs.google.com/presentation/d/1vr0eCQNvd0BeBuuVN8oOWnULqdBJ4C_f6581YHIMHz8/edit?usp=sharing))

      • Note: HDF5 doesn’t provide parameters

      • SS: distinction between filter & compressor?

      • DH: is SHUFFLE a compressor? (HDF5 doesn’t distinguish)

        • Storing last filter as the compressor (for now)

      • MD: list of the differences? DH: zfp e.g. isn’t defined. But not a large set.

        • John: https://pypi.org/project/hdf5plugin/

        • MD: interest also for fsspec to have the codecs matched

          up.

        • J-K: what’s the use case? DH: interoperability.

        • https://pypi.org/project/imagecodecs/ of interest (see

          converter)

        • TM: uses prefix of “imagecodecs:” in the codec id.

        • MD: couple of duplicates

      • General question of whether filter & compressor could be combined in V3 ?

        • MD: loose definition is filter(array) -> array and

          compressor(bytestream) -> array

        • John: history - sometimes people want to store text data

          (unicode) or arbitrary python objects

        • SS: so filter is exclusively for Python? For some,

          definitely.

      • Tobi: many people creating datasets with incompatibility issues

        • There should be an easy way to see how compatible a

          dataset is.

        • Warning? etc.Something implementations should inspire

          to.

    • Martin Durant: Who wants to give a quick (10min) overview of

      zarr at Dask Summit/IO workshop? https://summit.dask.org/

      • Organizing workshop “Data IO” 2 hours of talking
      • TileDB will be there as well.
      • Open informal. Ok to repeat.
      • Trevor: adapting files for Zarr (filesystem reference)
      • Josh/Jackson: at Life sciences
      • Stephan: Davies?

  • Finishing up:

    • Anyone want adding to the calendar invite? email: