2024-05-16
Attending: Dennis Heimbigner (DH), Sanket Verma (SV), Josh Moore (JM), Jeremy Maitin-Shepard (JMS)
TL;DR:
The meeting discussed the release of Zarr-Python 2.18.0, a new blog post by Joe Hamman, and updates on the sharding support in the R implementation. They also covered the implementation of manifest storage transformers, standardizing URLs for Zarr, and the removal of implicit groups in Zarr-Python V3.
Updates:
Meeting Minutes:
- Zarr-Python 2.18.0 out now: https://github.com/zarr-developers/zarr-python/releases/tag/v2.18.0
- One of the last few releases for Zarr Spec 2 - if there’s anything you want to get in, please reply/tag us in the PRs/issues
- New blog post by Joe Hamman: https://zarr.dev/blog/zarr-python-v3-update/
- Zarr-Python developers meeting new schedule - check here: https://zarr.dev/community-calls/
- Lachlan Deakin added support for sharding in his R implementation: https://github.com/LDeakin/zarrs/releases/tag/v0.13.1
Open agenda (add here 👇🏻):
- JM: In sharding you can recurse and browse through the chunks - somthing like chunks([x, y])
- DH: Treating sub-chunks as regular chunks - like what we decided during the storage transformers proposal -
- DH understands this proposal better and favours it
- The relevant issue: https://github.com/zarr-developers/zarr-specs/issues/220
- DH: Time to gets hands dirty with the implementation and figure out any problems we have
- JMS: Using storage transformers and codecs in Neuroglancer to achieve sharding
- DH:
- DH: Treating sub-chunks as regular chunks - like what we decided during the storage transformers proposal -
- SV: Manifest storage transformers - https://github.com/zarr-developers/zarr-specs/issues/287 - defines and implements on top of the storage transformer in V3 core spec - discussion 👇🏻
- JMS: Good to define the
JSONand add other formats later on - DH: FSSPEC interprets the URL in Kerchunk
- DH: Having complete key values in URL would help in the long run - DAP made a mistake earlier and we fixed it - having a complete URL is a better option and you can replace the contents within it later on
- JM: Having a complete URL in manifest storage transformer for Zarr would help us but there’s a question of backward compatibility
- JMS: https://github.com/zarr-developers/zeps/pull/48/ standardise the URL
- DH: URL spec defines the format and correct way of defninig a URL - if you consider things other than FSSPEC you should have a more standardised URL
- DH: Conforming to the URL Spec should be avoided actively
- JM: Having URL defined in the storage transformed would help - currently not defined
- JMS: Good to define the
- Fix typo - https://github.com/zarr-developers/zarr-specs/pull/294 - MERGED
- Updated Zarr-Specs license to CC-BY-4.0 - https://github.com/zarr-developers/zarr-specs/pull/295
- Implicit groups removed in Zarr-Python V3 via https://github.com/zarr-developers/zarr-python/pull/1827
- Corresponding PR in Zarr-Specs - https://github.com/zarr-developers/zarr-specs/pull/292