2023-06-15

Attending: Sanket Verma (SV), Alan Watson (AM), Jeremy Maitin-Shepard (JMS), Josh Moore (JM), Norman Rzepka (NR)

TL;DR:

The meeting highlighted SV’s recent talk on Zarr at Vrije Universiteit Amsterdam, providing access to the slides and code. Discussions covered the existence of command-line tools for Zarr, including the discovery of zarr-tools. AW shared Zarr adoption in a game project and encouraged its use in brain conferences. Additionally, there were updates on Allen’s utilization of Zarr, a forthcoming V3 blog post by SV, and ongoing discussions about addressing the fallback data types issue and the review timeline for ZEP0002.

Updates:

• SV gave a talk on Zarr @ Vrije Universiteit Amsterdam - https://vu-nl.libcal.com/event/4030713
  • Slides and code: https://github.com/MSanKeys963/presentations/tree/main/vu_amsterdam_6_23

Meeting Minutes:

• Are there any command line tools for Zarr?
  • Found this: https://github.com/BaroudLab/zarr-tools
  • JM: Was working on something on this but didn’t use it much
  • NR: There are bunch of tools which you can use OME-Zarr, having something like H5LS would be cool
  • JMS: Operations for small things makes sense - rechunking, copying
  • JM: Nextflow - workflow engine
• AM: BIL is working on games
  • https://www.mozak.science/landing
  • Pushing them to use Zarr for their image (.png) data
• AM: Interest and benefit in attending brain conferences - someone from the Zarr community
  • https://brainmeeting.swoogo.com/2023/home
• AM: Allen is using Zarr for their work
  • SV: Recent paper out of Allen: https://www.biorxiv.org/content/10.1101/2023.05.22.541700v2
  • Extensive usage at Allen have revealed some problems and it may be worth addressing them
  • JMS: Writing electrophysiology data in Zarr rather than tiff is a good oway to go forward
• SV: Blog post on V3 coming out soon!
• JMS: Fallback data types issue needs to be addressed
  • JMS: Not clear how it’ll be specified
  • JMS: How do you handle it in Zarrita?
  • NR: Currently, we do not!
  • SV: How serious it is? Implementation or spec issue?
  • JMS: Kind of ignoring for it now! We’re in implementation phase now!
  • JM: If everybody is ignoring it, then it’s fine
  • NR: Would not be straight forward to add it later!
  • JMS: If implementation doesn’t support, it’ll fail
  • SV: Have you started on the implementation?
  • JMS: Tensorstore has V3 minus sharding; planning to work on it this week
• NR: Rust implementation of V3 - https://github.com/clbarnes/zarr3-rs
• NR: Benchmarking in Zarrita
  • https://github.com/scalableminds/zarrita/blob/async/tests/test_perf.py
  • SV: Benchmarks in Tensorstore?
  • JMS: Seen bottlenecks in IO layer than array layer
  • NR: What about codecs?
  • JMS:
  • NR: 10x performance improvement would be great!
• NR: ZEP0002 review timeline
  • SV: V3 blog post, feedback for ZEP process, and then we can add ZEP0002 in the pipeline
  • NR: Ok!
  • SV: Maybe we need to invite Chris and others from the ZIC to the ZEP meeting
  • JM: There used to be a Zarr-Rust meeting