2024-03-07

Attending: Sanket Verma (SV), Ward Fisher (WF), Davis Bennett (DB), Josh Moore (JM), Thomas Nicholas (TN), Jeremy Maitin-Shepard (JMS)

TL;DR:

The meeting discussed enhancing Zarr store browsing, pushing Kerchunk functionality into Zarr, and the potential for a chunk manifest as a ZEP. Additionally, the group explored ideas for revising ZEP0, creating a Zarr specification IETF standard, and shared updates on upcoming Zarr HTTP Extension and SEA conference events.

Updates:

Meeting Minutes:

  • TN: Jed wants to have nice ways to browse the Zarr stores - they have nice ways to browse .tiff files already
    • Wants to propose an extension to add more information in the metadata
    • The end result would look more like a Xarray HTML wrapper
  • TN: https://hackmd.io/t9Myqt0HR7O0nq6wiHWCDA?view
    • Had a conversation with folks over at Development Seed, NASA, Earthmover
    • DB: Pushing Kerchunk functionality into Zarr stores
    • DB: Whether the feature could be file format agnostic?
    • TN: Argues that it should be a ZEP - and can be read every Zarr implementation
    • JM: Having same thing implemented in FSSPEC
    • DB: Would ZEP
    • WF: HDF5 group may be open to a conversation
    • SV: https://zarr.dev/zeps/meetings/2023/2023-08-10.html might have some useful information
    • TN: recaps the conversation for JMS
    • TN: Should concatenation be a part of the current ZEP?
    • DB: Any reason you don’t want to concatenate HDF5 and other file formats?
    • TN: Chunk manifest would point inside the arrays - chunk manifest could let you create a Zarr store over other formats as well
    • DB: This would make Zarr as an API/access pattern
    • TN: Can be created and tested fairly separate to Zarr - personally think chunk manifest is neat feature - implementation can support/not support it
    • DB: Array mutation can break the concatenation - having guidelines for archival arrays would help
    • TN: Currently we’re thinking about read-only case
    • TN: Virtualisation in Kerchunk is a spotlight feature
    • JMS: Manifest is a good idea and keeping it separate would be a minor difference - needs to align with Kerchunk
  • JM: report/ZEP idea (time permitting)
    • https://w3id.org/ro/crate/
    • JM: Putting ro-create inside Zarr - or making Zarr specification a IETF standard
    • JM: Would probably go ahead and write a convention in NGFF space
    • https://fairdo.org/
    • JM: Have a mechanism for going up/down the hierarchy - useful for the HTTP extension discussions
  • Revising ZEP0
    • https://github.com/zarr-developers/zeps/pull/59 - comments/feedback welcome
    • DB: :+1:
    • DB: Would be easy to have a single PR for my ZEP
    • JMS: Putting narrative document in PR description
    • JM: Weird for commenting on the PR description and for the public visibility
    • JMS: Rationale can be put down as a footnote
    • JMS: Having numeric numbering is something Python follows
    • JMS: The actual specification change can also serve as a ZEP narrative
    • SV: We can pick out certain sections out of the ZEP narrative document
    • JMS: Having a PR template similar to ZEP’s narrative could also help us
  • WF: https://sea.ucar.edu/conference/2024
    • In-person and virtual registrations are available