2023-08-10

Attending: Sanket Verma (SV), Josh Moore (JM), Jonathan Striebel (JS), Norman Rzepka (NR)

TL;DR:

The meeting discussed updates on Zarr-Python working groups, a proof-of-concept implementation for ZEP0003, and presentations at SciPy 2023. Notable mentions included removing watermarks from the V3 spec, recognizing contributors for ZEP0001, and ongoing discussions on collaborating with Blosc and Dask for efficient chunking and sharding across the PyData stack. There was also exploration into a lightweight ZEP process for codecs, with considerations for updating ZEP0000 and scheduling ZEPs for voting. Fundraising discussions and potential collaborations with Napari were explored to sustain momentum in the Zarr project.

Updates:

Meeting minutes:

  • JM: JS are you using Zarr at work?
    • JS: We’re convinced that it’s a good idea to use Zarr! 😄
    • Growing rapidly - good thing
  • SV: Presentation for EuroSciPy 2023!
    • JS: Coming up
    • SV: You can also cite SciPy 2023 talks
    • JM: Will be using the EuroSciPy 2022 poster for an upcoming meeting
  • SV: Tweeting about the contributors for ZEP0001 and tagging everyone
    • JM: Thanks to everyone! 🙏🏻
  • JS: Removes the watermark from the V3 spec https://github.com/zarr-developers/zarr-specs/pull/260
    • Figured out the CSS selector
  • JM: Zeiss got back to JM
  • SciPy 2023 discussions
    • James Webb Space Telescope - how they can use Zarr
    • JS: Met with Francesc (Blosc) ?
    • JM: Yeah, we spend a lot of time and it was great!
    • How Blosc and Sharding can co-exist together - like a package
    • SV: Sharding can provide cloud enability to Blosc2 - discussions with Francesc
    • JM: Recently filled out feedback for CZI EOSS - we can do join grants as well
    • NR: Writing a Blosc2 codec for Zarr could help
    • JM: Dask chunking comes down to .chunk() property for the object - how about data API chunking specification around the chunks? - chunking could work across the whole PyData stack - and we can add sharding too - could help with what’s the efficient access pattern for sharding chunk!?
    • Unified package for Zarr (Sharding), Blosc2 and Dask and other packages
    • NR: Interesting! Sharding has 2 access pattern
      • Chunk level for read and shard level for write
      • For Dask purposes you probably want the shard access pattern - because you’re in a HPC environment
    • JS: Writing it as a spec and collaborating with Dask and Blosc team
    • NR: Agreed!
    • JS: Sharding also needs a lot of explanation - lots of education needed
    • NR: Limbo state rn - blog posts and videos can help a lot - maybe after 6 months
    • JM: Unifying names - block.dev? - and same documentation as well
    • SV: Can include HDF5 as well
      • JM: HDF5 could be beneficial if you’re working on cluster/HPC and Zarr can help you bring that data down from the cloud to your machine
    • NR: Can apply to EOSS grant with same applications
    • JM: Less chances of getting funded
    • JM: Zarr can solve world hunger! 😁
    • NR: Good momentum but need to deliver as well
    • JM: Zarr as a sister project of Napari!?
    • SV: Having conversations regarding fundraising for Zarr to keep the project funded
    • We can work on joint grant or something similar
  • NR: Lightweight ZEP process for codecs?
    • JM: Light voting procedure?
    • NR: Could be!
    • JM: A new ZEP to update ZEP0000 and add a new type of ZEP in the types and loosen the restrictions
    • NR: No problem with voting but how do we setup the ZEPs up for voting - anyone can do it via creating a issue but that can lead to mayhem if everyone starts doing it!
    • JM: We’re following a chronological order but we can have a statement which can allow lightweight ZEPs to come in while big ones are on the way
    • Having a separate ZEPs for codecs and extension with less voting burden
    • How do we schedule the ZEPs for voting?
    • Something for https://github.com/zarr-developers/zarr-specs/pull/256
    • JM: There are improvements we can make to ZEP0000 and let’s keep working on that