Attending: Sanket Verma (SV), Norman Rzepka (NR), Ward Fisher (WF), Jeremy Maitin-Shepard (JMS)


The meeting introduced new ZEPs by Davis Bennett (ZEP0006) and Isaac Virshup (ZEP0007). Discussions included the possibility of hosting Zarr-Con with NASA F.15 grant support and updates on ZEP0002 progress, particularly in Tensorstore implementation. The addition of ZipStore as a ZEP was explored, with considerations for conventions, URL structures, and potential contributors from the microscopy and napari communities. Additionally, a discussion on handling non-Zarr stores by Zarr and the need for defined behaviors in case of malformed data was addressed.


Meeting Minutes:

  • WF: NASA F.15 grant could help hosting Zarr-Con over at Unidata
  • SV: Update on new ZEPs by Davis and Isaac
    • NR: Overview of the ZEP0007
    • WF: Character encoding addressed? - Not implemented robustly across NetCDF
    • SV: Norman as co-author?
    • NR: No, just left some comments
    • JMS: Define a name for the codec - array to bytes - can be applied to raw data buffer
    • NR: Could model it as a data type - not clear how the translation from bytes would work in a codec
    • JMS: Encourage a spec PR first - make things straightforward
    • SV: ZEP document and spec PR - anyone can come first - depends which is the clear and straightforward way to introduce the changes
  • SV: ZEP0002
    • JMS: Extremely close on tensorstore
    • NR: Zarrita.js can be added to the sharding implementation in the issue review
  • NR: Adding ZipStore as a ZEP
    • JMS: Added read-only support for ZipStore to tenstore
    • NR: Certain features that can be included in the ZEP - like allow different types of hierarchy
    • JMS: Various ways to use ZipStore in Zarr-Python - depends on different ways how want to organise your data in the Zip
    • NR: Maybe more of a convention - Zip on S3: How do you access it? (URL gets funky)
    • JMS: s3://bucket/path/to/zip.zip|zip:path/to/array/|zarr3 - pipe URL - convey what’s happening - : downsides - they’re valid in a URL
    • NR: Go down further and address things further in the Zarr like array
    • WF: We have Zip support in NCZarr - not the similar URL style
    • NR: Microscopy folks - napari folks - Saalfeld could be potential people who could work on the Zip ZEP
    • NR: Protocol for Google storage? GS or GCS
    • JMS: gs://bucket/path - GS
    • JMS: General Zip required sequential access
    • JMS: Standardizing some kind for URL would be good thing
    • NR: Getting feedback from Ward, Stephan Saalfeld would be good
    • WF: HTTP post style syntax in NetCDF is supported
  • WF: What would happen if we try to read non-Zarr store by Zarr?
    • JMS: Looking at the metadata file and then figuring it out?
    • WF: Some part of NetCDF uses HDF5 and we try to open it with Zarr and it crashed
    • WF: Curious to what the failed open() should look like? Having a defined behaviour would be good
    • JMS: Launch missiles if the data is malformed! 😄🚀
    • WF: NetCDF have certain error code when it can’t read insted of crashing the software - should be a part of the spec
    • JMS: Could be a good addition
    • WF: Just curious about the crashing!