2022-08-10

Attending: Sanket Verma (SV), Josh Moore (JM), Jeremy Maitin-Shepard (JMS), Alex Merose (AM), Brianna Pagán (BP), Hailey Johnson (HJ), Hailiang Zhang (HZ), Jonathan Striebel (JS), Martin Durant (MD), Norman Rzepka (NR), Shivank Chaudhary (SC), Ward Fisher (WF), Mahabal Hegde (MH), John Kirkham (JK), Isaac Virshup (IV)

TL;DR:

• Introductions w/ favourite sports. ⚽️🧗🏻🥊
• ZEP1 & ZEP2 are open for review and feedback! 🎉
• Browse https://zarr.dev/community-calls/ for previous meetings notes. ✌🏻
• 2.13.0a1 releasing soon with updates from Davis, Mads R.B. and Jonathan 🫡
  • Josh: 2.13 next alphas
  • https://zarr.readthedocs.io/en/latest/release.html#release-2-13-0
• async-zarr by Martin Durrant: https://github.com/martindurant/async-zarr → Contributions welcome! 🙌🏻
• About SPEC V3 by Alex: Could a video codec be implemented!? (Interesting use-case) 🧐
• Hailiang, Briana, Mahabal from NASA-Goddard are planning to write a ZEP for extension. 🤝🏻

Meeting Notes

Introductions:

• Favorite sport!
• Feel free to add links to your work here

Updates:

• ZEP1 & ZEP2 are open for feedback!
  • Review under https://zarr.dev/zeps/draft_zeps/
  • Comments on https://github.com/zarr-developers/zarr-specs/pulls
• Browse https://zarr.dev/community-calls/ for previous meetings notes
• Jonathan Striebel is presenting poster on Zarr @ EuroScipy 2022. If you’re attending the conference, please say, Hi 👋🏻!
• 2.13.0a1 releasing soon with updates from Davis, Mads R.B. and Jonathan
  • Josh: 2.13 next alphas
  • https://zarr.readthedocs.io/en/latest/release.html#release-2-13-0
• Phase 1 of GSoC 2022 completed! 🎉 Check progress here
  • JMS: would be good to have specification (json-schema?) for each
  • Goal: have clients interact with the registry to give users info/feedback
• async zarr https://github.com/martindurant/async-zarr
  • anacoda hackweek per quarter (2-day-hack)
  • for discussion https://gitter.im/zarr-developers/community?at=62f3ed24d020d223d36587d5
  • http only and other simplifications
  • JMS: targeting runtimes outside of the browser?
  • AM: cloudflare worker? WebAssembly support. (MD: already in pyodide)
    • lightweight VMs (e.g., for security)
  • https://github.com/zarr-developers/community/issues/14
  • IV: story for downstream library developer to use? rewrite to use await
    • MD: definitely must use await (can’t go in and out of the event loop)
    • use case: first chunk of several arrays
    • MD: e.g. what would xarray for example use it
    • MD: somethings already work: bokeh, etc.

Open agenda:

• …add here …
• AM: Question about the Zarr Spec v3 (ZEP1)
  • Like that it’s very bare bones
  • Though experiment: Could a video codec be implemented?
    • Compress across time; key frames
    • JMS: 3d xyt chunks would work (individually)
    • MD: variable length chunks. Critical to video compression (per key frame)
    • MD: if each chunk has all the time points
    • MD: but also in favor of variable length chunks
    • JMS: what’s the connection?
    • MD: video compressions support large range of chunk size based on how quickly the video is changing
    • JMS: just make time chunking big enough and internal is a detail
    • JS: remove chunking across the time dimension
      • currently inefficient, but with partial reads it could work
      • let video codec request chunks of data from the store
    • MD: internal to codec or explicit at the zarr level
    • JM: difference in the fundamental model? (atomicity)
    • MD: 1-dimensional delta codec, make it across an arbitrary dimension?
    • AM: if Zarr intends to be the metadata format, this is a stress test.
    • JMS: with fixed key rate,
    • JM: see also https://mpeg-g.org/
    • 
• HZ: extension proposal
  • implementation for multidimensional data analysis
  • introducing auxillary datasetse in reduced datasets (non-scalar, accumulation value)
  • helps to speed up computation. Ryan A. suggested a spec extension
  • MH: averaging over time or spatial extensions
  • JM: cf. https://github.com/zarr-developers/zarr-specs/issues/50
  • IV: perhaps like transforms https://github.com/ome/ngff/issues/101
  • JS: difference of whether or not it leads to an additional array
• IV: Non-uniform chunks – timing?
  • conversation with JK at SciPy
  • broad desire to have them exist. any objections?
  • have several masters students to put on this
  • JM: ZIC?
  • IV: can discuss if in spec or as an extension
  • JS: would still have a formal spec even if an extension (eases adoptions, clear interface)
  • IV: ZEP0001 timeline?
  • SV: on me. working with Alistair to apply the modifications. ASAP.
• JMS: meta-issue for scheduling time to work on the V3 spec
  • way to speed up progress? additional meetings?
  • SV: ZIC meeting?
  • JM: editorial meeting? Add JMS?
  • JS: happy to be in discussions with AM but also open issues that need discussion