2024-05-02

Attending: Josh Moore (JM), Sanket Verma (SV), Ward Fisher (WF), Jeremy Maitin-Shepard (JMS), Thomas Nicholas (TN)

TL;DR:

The meeting discussed the upcoming Zarr V3 release candidate, status and integration of the chunk manifest ZEP, and potential revisions for combining ZEPs. They also covered progress on ZEP0 and plans to move ZEP2 to “Active” status, while tabling the discussion on removing implicit groups.

Updates:

Meeting Minutes:

  • WF: Dennis has a PR coming up to revise the Zarr V3 - 6 months long work - release candidate coming soon!
  • TN: Status of unfinished ZEP w.r.t. to chunk manifest?
    • E.g. Have a sharded chunk manifest?
    • JMS: Chunk manifest could refer to entire shard - use case might not be clear
    • JMS: For viz tools you would not load entire shard at once
    • TN: Changing chunk manifest ZEP or accomodate chunk manifest in sharding codec?
    • JMS: Not really need to change
    • JM: Maybe there’s a way to re-write the ZEP in a way which the existing ZEPs are composable - basically how extensions would interact with each other
    • TN:
    • JMS: There might be cases where combination of codecs may not work well
    • TN: Combining codecs seems straightforward compared to variable chunking which specifies what is allowed and what not
    • JMS: The proposal which changes the data model are tricky - wanted to add non-zero origin
    • TN: Interested in variable chunking
    • SV: Would you in be interested in contributing to ZEP3?
    • JM: We could also start thinking about ZEP2+ZEP3, ZEP3+ZEP4, ZEP4+ZEP2…
    • JMS: Any reason for not using Kerchunk?
    • TN: Chunk manifest is clearly defined Zarr store compared to kerchunk (which kinda looks like Zarr) - reference file-system are not defined - there’s value of getting chunk manifest into Zarr specification as Kerchunk is more than Zarr
    • JMS: The actual implementation of the file-system would be same across the various libraries
    • TN: Relying on single maintainer code is not an ideal situation
    • SV: ZP V3 implementation was outdated which led to creation of Zarrita and then finally re-using Zarrita for ZP V3 refactor
    • TN: Working on nit-picking Xarray for Virtuali-Zarr
    • TN: Zarr arrays are kind-of lazy arrays - when you index into Z-arrays they provide you with bytes not the actual Zarr arrays - Xarray has lazy-loading hidden inside in codebase and there has been discussion to make it a standalone library
    • JMS: We could have two sizes for chunks - stored size and actual size for variable chunking strategy
  • Move ZEP2 from Accepted to Active
    • JM: Would be good to move ZEP1 and ZEP2 both at the same time
    • SV: ZP V3 refactor would be a good time to move ZEP1 to active
  • Finalise ZEP0 revisions
  • TABLED