2022-11-03

Attending: Josh Moore (JM), Jonathan Striebel (JS), Jeremy Maitin-Shepherd (JMS), Sanket Verma (SV)

TL;DR

Discussions were held on how to move forward with ZEP1 quickly. The summary can be viewed here. Then the attendees discussed extensions in V3, and JMS is considering trying with non-zero origin. SV joined the meeting after 30 mins. After that, JS mentioned some high-level issues looming around V3 spec.

Meeting Minutes:

  • JMS: number of PRs that could be merged into the working draft
    • JS: don’t want to just close it
    • JM: can we cross link e.g. JMS’ PR? Yes.
    • ==> once all cross-linked close PR.
  • JS: when to merge?
    • JM: when it matches the consensus?
    • JS: ok, but don’t have merge rights.
    • ==> Let’s merge proactively.
  • see: https://github.com/zarr-developers/zarr-specs/pull/149#issuecomment-1302440391
  • extensions
    • JMS: thinking of trying with non-0-origin
    • JM: think that’s a general principle we should try for all issues/PR is “could it be an extension”
    • JMS: thinking of extensions as plugins? Not exactly.
    • JS: how to influence if an implenentation adopts an extension? if there’s a concrete implementation / clear interface
    • JMS: agreed and some obvious ones (codecs) but not clear there will be a broader abstraction
      • JS: “index transformer” perhaps
      • or as transformer if multiple of chunking
      • JMS: unfortunate limitation
    • JMS: re: transformers - it doesn’t make sense to compose a different storage transform before sharding
      • JS: depends. cache of chunks or shards? also checksum
      • JM: codec is similar
      • JMS: caching enabled in code, but not in zarr metadata
      • JS: that’s in spec, yes. “runtime-only” but still before or after
      • JMS: when implementing sharding, would check if it’s first
        • want to be able to tell the user “this is the graularity to write”
        • JS: good that it’s flexible. like c/f order.
        • JS: mention in implementation “sharding must be first”
      • JMS: composing makes for useful extension point
      • JS: most important point: are we sure enough about the extension points?
  • Sanket joins 🧑🏻‍💻
  • Jonathan: high-level issues looming
    • paths/URL discussion (needs an issue)
    • global transformers
    • variable chunk length (possibly origin offset)
    • indexing more abstract
    • upgrade path! ({“extension”: [“@v2-layout”]})