10th February, 2021

Attendees: (Add your own): Josh Moore (OME), Matthias Bussonnier, Jackson Brown, Ralf Gommers, Ryan Abernathey, John Kirkham, Ryan Williams, Chris Barnes, Andrew Champion

  • Greetings from Alistair! (Josh)

  • Quansight check-in (Ryan, Ralf, etc.)

    • Extension until end of March (40k to consume)

    • Matthias: few hours per week (hard for someone else to ramp up)

      • Few PRs to rebase and get in

    • Ralf: few other people who could jump in.

    • Various Java efforts (nczarr-c, JZarr, n5)

    • Ralf: is Matthias blocked on feedback

    • Get on a call with Alistair

    • Final report May 1

  • Community

    • Rust & Zarr implementation interoperation

      • Chris: Status. rust-pyn5 bindings around rust-n5. h5py_like-compatible. Designs on other chunked backends. Zarr v3 (~maybe v2). In a separate repo, but Andrew is driving that.

      • Low-level repo has a portion with zarr v3 as of a couple of months ago.

      • Issues with flexibility with the protocol. Didn’t meld with rust idioms. (Knowledge of components) (up and downstream)

      • Matthias: numcodec implementation? All is handled in rust. Based on available crates. i.e. only support a subset. (Not all implementations are great)

      • As much as possible in rust. Attribute metadata access is in python.

      • See also n5-wasm (also neuroglancer pre-computed). Interop between javascript and wasm is difficult (handling of ndim arrays)

      • Matthias: Cryptography

      • John: which use cases for rust? currently being deployed on both ends of image browsing. On server to read n5 volumes into jpeg tiles on the fly. In the browser for a tool that looks at a lot of jpeg/n5 files. Nothing super-high through-put.

      • (Andrew joins)

      • Josh: potential for sharing an implementation & wrap. Sure. At least as much as C. Not great for introspection. But pretty versatile.

      • Josh: v3 as a subset of n5? AC: zarr does more than n5-use cases need. (Need to work with KLB and other formats) Zarr := uniform interface to similar emerging specs but also be plastic & versatile (extensibility)

      • AC: Rust doesn’t have a great dynamic linking story. Not sure how a python interface on top of rust is going to work.

      • MB: makes sense that extension are at compile time. That’s why they should all be listed in the top-level metadata.

      • JK: 2 things entangled

        • If I make an implementation, do I add the functionality

        • Do they live in one binary or in multiple.

      • CB: data communication to user whether or not its readable

      • nczarr? Can use rust library. Rust library can’t use C library due to wasm exporting.

      • AC: best implementation to work against? Zarrita or xtensor-zarr

      • Josh: Assume https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/ is the working version.

      • AC: working version is in *https://github.com/sci-rs/zarr *

      • JK: Spec issues tracked here: https://github.com/zarr-developers/zarr-specs/issues

  • Core