2023-11-15

Attending: Sanket Verma (SV), Justus Magin (JM), Josh Moore (JM), Ward Fisher (WF), Norman Rzepka (NR), Dennis Heimbigner (DH), Davis Bennett (DB), Jeremy Maitin-Shepard (JMS)

TL;DR:

We started the meeting with introductions and asking about the favourite movie prop everyone wants. There are some good answers there! 💍🗡️🖼️

After this, SV proposed changing the format of the meetings and inviting speakers for quick presentations. JMS asked if there were any changes required for ZEP0002. We briefly discussed the changes after the ZEP0002 final voting. DH initiated a discussion on consolidated metadata towards the end of the meeting.

Updates:

• Zarr-Python
  • Zarr-Python 3.0 Roadmap - https://hackmd.io/0DVKP6d9QI-VaHc0zvOuxw
  • Benchmarking & performance group updates
    • Benchmark results for small random reads from a fast SSD using fio - https://github.com/zarr-developers/zarr-benchmark/discussions/26
    • Comparison of Tensorstore and Zarr-Python benchmarks - https://github.com/zarr-developers/zarr-benchmark/discussions/25
• Cancel community, ZEP, and other meetings around Christmas - thoughts?
  • 👍🏻: all around for Week of 26th December
• NR: ZEP0002 was approved! We baked a cake. 🎂
• JM & DB: https://forum.image.sc/t/outcomes-of-the-next-generation-bioimage-analysis-workflows-hackathon/88733/3

Meeting Minutes:

• Introductions; props
  • SV: Rings from LOTR
  • WF: Artwork from The Royal Tenenbaums
  • DH: ‘The House’ from the movie Haunted
  • Justus M.: Xarray maintainer; prop - Mjolnir
  • NR: Andúril, Aragorn’s sword from LOTR
  • JM: Briefcase from the movie, “What’s up, doc?”
  • DB: Dune’s Lasers
• Ideas for changing the format of community meetings?
  • Invite speakers from various domains for quick 10 minutes presentations - followed by 5 mins. QnA
  • More ideas?
• JMS: What other ZEP0002 changes are required?
  • JM: Have intermediate votes
  • NR: A small number of implementations should support a ZEP, i.e. Zarr-Python
    • A dedicated time for veto in the start of the voting phase
  • JM: Having a period of silence should not be there - also updating the active implementations
  • JMS: Zarr-Python will be using Zarrita
  • NR: For sharding the V3 support makes sense but not for other ZEPs
  • JM: Change ZEP0000 and then open ZEP3&7 for voting - also get clean Zarr-Python maintainers list
  • NR: Taking Zarrita as parts and using it for Zarr-Python
  • NR: Doesn’t plan to maintain Zarrita for long term
  • JMS: In order for test implementations it needs to be maintained
• DH: Status: 2/3rd completed for V3 implementation in NetCDF
  • JM: Thoughts on sharding?
  • DH: A bit surprised on the codec implementation - mostly a caching issue
• SV: The changes after ZEP2 voting
  • NR: index_location is quite small change
  • NR: What about ZEP1 changes?
  • JM: Trevor is maintaining Zarrita.js and Anderson (CarbonPlan) will be getting rid of Zarr-js
• JM: Maintain a list of active implementations
• Justus M.: Ryan held a Zarr BoF at a conference
  • https://hackmd.io/0tCWY-MAT5Ob0x1BbDKBHA
• DH: Aggregate(consolidate) metadata issue for V3
  • NR: ZOM (Davis’ ZEP) could be one way for this - so far no champion at the moment
  • DB: Trying to put a connection to consolidate metadata
  • JM: As a secondary thing?
  • DB: This is collorary
  • JM: People will care more about consolidate metadata than ZOM - DH raised an issue back in 2021: https://github.com/zarr-developers/zarr-specs/issues/113
  • DB: Can work with the consolidated metadata - but would not go in the nitty gritty details
  • JMS: Would be good to champion the consolidated metadata who actually cares about it
  • NR: Yeah, I think we need someone who can drive that - and someone who can remove redundancies - if you’re in root group you have everything in there and array will still have Zarr’s JSON and that would be redundant and out-of-sync - a typical cache problem
  • DH: Concern being tradeoff - everything in single place - good for cloud - have datasets with enormous metadata sizes and cannot do lazy loading of the metadata - walking as necessary helps us - HDF5 version does infact lazy loading
  • JMS: Single array with bunch of attributes?
  • DH: Mostly because of cf conventions - general problem
  • DH: Option for moving attribute in a different object - out of zarr.json -
  • JMS: Binary formats for Zarr metadata that could support partial read
  • DB: SQLite database for Zarr metadata
  • DH: Considered the option - Google protobuf implementation - contemplated for DAP4 - a compact implementation - don’t know if other have thought about the other formats -
  • JMS: Protobuf doesn’t support random access
  • NR: Discoverability is an issue
  • JM: Listing all the metadata in the group could be good starting step
  • DB: 💡Idea💡 Array within array - express dependency relationship b/w arrays