2024-01-24

Attending: Ward Fisher (WF), Sanket Verma (SV), Davis Bennett (DB), Gābor Kovācs (GB), Joe Hamman (JH), Jeremy Maitin-Shepard (JMS)

TL;DR:

Highlights include updates on Zarr activities, such as a sprint in NYC, progress on Zarr-Python refactor, and discussions on codec compatibility and metadata changes for Zarr-Python V3, alongside introductions with favorite foods and considerations for Numcodecs usage and backward compatibility.

Updates:

Meeting Minutes:

  • DB: Will be opening a new PR which is currently in a new branch
  • Release: Would like to spread the word about dropping Python 3.8
  • WF: AMS presence taking place in Baltimore
    • JH: Ryan and Joe are going be there
  • Introductions w/ favourite food
    • SV: Dumplings
    • JH: CTO at Earthmover; burritos/pizza
    • DB: Independent software contractor; spicy food
    • GB: Research Scientist @ Allen Institute; hungarian sweet
    • WF: Technical team lead @ Unidata; eggs and rice/chicken tikka masala
  • JH: Numcodecs in the Zarr-Python world
    • JH: Zarr-Python V3 will have plugin interface (array→bytes, array→array, bytes→bytes) - There is no array→array codec in numcodecs - In V2 codecs live in numcodecs
    • JH: What packages are using numcodecs only?
    • JH: Maybe they’re using through Zarr!
    • DB: Adding a ABC codec array bytes
    • JH: Reached out to John, still waiting to hear from him
    • JMS: Downloads for Zarr-Python and numcodecs are good - we should not introduce a backward-incompatibility
    • JMS: Originally designed to have codecs for Zarr - keep numcodecs for backward compatible packages like V2
    • JMS: Changes in metadata b/w V2 and V3 which can be troublesome
    • JH: https://github.com/zarr-developers/numcodecs/pull/501
    • JMS: New design depends on what things you want to pass through the codecs
  • SV: Martin might be showing up
  • SV: Endorse SPEC0
  • DB: Spec has different type of codecs - is typing based on the signature
    • JMS: Implementation need to know what type of codec is it
    • DB: Looking at the properties and wondering to add a new type
    • JMS: Neuroglancer has a registry where we lookup for codecs; registries aren’t necessary!
    • DB: Creating the registry while adding new codecs can be useful
    • JMS: Example Neuroglancer: https://github.com/google/neuroglancer/blob/master/src/datasource/zarr/codec/resolve.ts
    • JH: Would be interesting to get Norman take on this
    • JMS: The design is influenced because of multiple threads and you can only ship bare objects
    • DB: Burns to see type definitions in typescript! 😓
    • JMS: Python has poorly designed type definitions when compared to typescript
  • DB: Status of V3 in Neuroglancer?
    • JMS: Definitely pretty complete but have some caveats like handling transpose, etc.
    • JMS: Like handling sharding; probably as good as a source for V3 implementation as anything out there
    • JMS: Tensorstore is also complete; also more general than Neuroglancer
    • DB: Has bias coming from V2; so want to see how others are doing things in a new ecosystem
  • TABLED