2023-09-07

Attending: Sanket Verma (SV), Norman Rzepka (NR), Ward Fisher (WF), Jeremy Maitin-Shepard (JMS)

TL;DR:

The meeting introduced new ZEPs by Davis Bennett (ZEP0006) and Isaac Virshup (ZEP0007). Discussions included the possibility of hosting Zarr-Con with NASA F.15 grant support and updates on ZEP0002 progress, particularly in Tensorstore implementation. The addition of ZipStore as a ZEP was explored, with considerations for conventions, URL structures, and potential contributors from the microscopy and napari communities. Additionally, a discussion on handling non-Zarr stores by Zarr and the need for defined behaviors in case of malformed data was addressed.

Updates:

• ZEP0006 by Davis Bennett: https://github.com/zarr-developers/zeps/pull/46
• ZEP0007 by Isaac Virshup: https://github.com/zarr-developers/zeps/pull/47

Meeting Minutes:

• WF: NASA F.15 grant could help hosting Zarr-Con over at Unidata
• SV: Update on new ZEPs by Davis and Isaac
  • NR: Overview of the ZEP0007
  • WF: Character encoding addressed? - Not implemented robustly across NetCDF
  • SV: Norman as co-author?
  • NR: No, just left some comments
  • JMS: Define a name for the codec - array to bytes - can be applied to raw data buffer
  • NR: Could model it as a data type - not clear how the translation from bytes would work in a codec
  • JMS: Encourage a spec PR first - make things straightforward
  • SV: ZEP document and spec PR - anyone can come first - depends which is the clear and straightforward way to introduce the changes
• SV: ZEP0002
  • JMS: Extremely close on tensorstore
  • NR: Zarrita.js can be added to the sharding implementation in the issue review
• NR: Adding ZipStore as a ZEP
  • JMS: Added read-only support for ZipStore to tenstore
  • NR: Certain features that can be included in the ZEP - like allow different types of hierarchy
  • JMS: Various ways to use ZipStore in Zarr-Python - depends on different ways how want to organise your data in the Zip
  • NR: Maybe more of a convention - Zip on S3: How do you access it? (URL gets funky)
  • JMS: s3://bucket/path/to/zip.zip|zip:path/to/array/|zarr3 - pipe URL - convey what’s happening - : downsides - they’re valid in a URL
  • NR: Go down further and address things further in the Zarr like array
  • WF: We have Zip support in NCZarr - not the similar URL style
  • NR: Microscopy folks - napari folks - Saalfeld could be potential people who could work on the Zip ZEP
  • NR: Protocol for Google storage? GS or GCS
  • JMS: gs://bucket/path - GS
  • JMS: General Zip required sequential access
  • JMS: Standardizing some kind for URL would be good thing
  • NR: Getting feedback from Ward, Stephan Saalfeld would be good
  • WF: HTTP post style syntax in NetCDF is supported
• WF: What would happen if we try to read non-Zarr store by Zarr?
  • JMS: Looking at the metadata file and then figuring it out?
  • WF: Some part of NetCDF uses HDF5 and we try to open it with Zarr and it crashed
  • WF: Curious to what the failed open() should look like? Having a defined behaviour would be good
  • JMS: Launch missiles if the data is malformed! 😄🚀
  • WF: NetCDF have certain error code when it can’t read insted of crashing the software - should be a part of the spec
  • JMS: Could be a good addition
  • WF: Just curious about the crashing!