2023-09-21

Attending: Sanket Verma (SV), Josh Moore (JM), Ward Fisher (WF), Davis Bennett (DB), Jeremy Maitin-Shepard (JMS)

TL;DR:

The meeting featured updates on new ZEPs, including ZEP0006 (Zarr Object Model) by Davis Bennett and its implementation progress, ZEP0007 (String) by Isaac Virshup, and ZEP0008 (URL Syntax) by Jeremy Maitin-Shepard. Discussions included an overview of the newly submitted ZEPs, with considerations for generalizing URL syntax and exploring query syntax similarities with JSON. Additionally, discussions touched on the need for standardizing JSON schemas for Zarr, the compatibility of Zarr shards with HDF5, and plans to kick off the voting process for ZEP0003, particularly its usefulness for Tensorstore.

Updates:

• ZEP0006 by Davis Bennett: https://github.com/zarr-developers/zeps/pull/46 (Zarr Object Model)
  • Implementation: https://github.com/zarr-developers/zarr-python/pull/1526
• ZEP0007 by Isaac Virshup: https://github.com/zarr-developers/zeps/pull/47 (String)
• ZEP0008 by Jeremy Maitin-Shepard: https://github.com/zarr-developers/zeps/pull/48 (URL Syntax)

Meeting Minutes:

• Overview of the newly submitted ZEPs
  • JMS: ZEP8 Could be generalised apart from the Zarr ecosystem - provides parameters at each specific level
  • DB: Considering query strings?
  • JMS: Clearly diverging from convential syntax -
  • JMS: Issue with # syntax - interpretation will be different - a few downsides of using it - argument for using fragment identifier
  • JM: Descending down the attributes in N5 land?
    • https://github.com/ome/ngff/issues/144#issuecomment-1314299599
  • DB: Query syntax like JSON - the idea was shared among and used across in N5 land
• DB: ZEP0006 (ZOM) discussions - Tally Lambert from Napari was looking for JSON schema for Zarr
  • JMS: JSON Schema for tensorstore
    • V2: https://github.com/google/tensorstore/blob/master/tensorstore/driver/zarr/schema.yml
    • V3: https://github.com/google/tensorstore/blob/master/tensorstore/driver/zarr3/schema.yml
  • DB: Wise thing to move towards the standardisation of the schema
  • JMS: Consolidated metadata
  • JM: Engage positively with consolidated metadata and not break it
  • JMS: V3 consolidated metadata not in-line with ZOM would be a bad thing :)
  • DB: Could be use with HDF5 as well and can be future proof for the upcoming Zarr specifications
• SV: Zarr shards as a valid HDF5
  • DB: Need to have a mechanism where both the formats can talk to each other - otherwise may lead to brittleness
  • JMS: The approach is hacky atm
• Martin wants to kick-off ZEP0003 voting ASAP
  • Prototype implementation: https://github.com/zarr-developers/zarr-python/pull/1483
  • Techincal review needed
    • DB will have at it soon!
  • JMS: Useful for Tensorstore