3rd June, 2020

Attending: Matthias Bussonnier, Ward Fisher (can only stay 30 min -20:21)m John Kirkham, Dennis, Ryan Williams, Alistair Miles, Martin Durant, Andrew Fulton, Josh Moore (20:18-)

Apologies:

  • Standing items

    • Introductions: new joinees should feel welcome to introduce

      themselves and their interest in zarr/n5.

    • Noone new.

  • Etiquette to ping for feedback/merge on pull-requests on various repositories ?

    • what kind of turnaround time should contributors/committers

      expect (for reviews, discussion)?

    • straightforward doc updates etc. can sometimes sit for O(weeks)

    • different repos likely to move at different paces (spec

      updates/discussions will go more slowly)

  • Partial Decompression discussion ?

    • numcodecs#235

    • cuts across lots of the stack

    • Alistair: Who and why do we want to use partial decompress.

    • Andrew: clients need partial read with network store.

    • Matthias potentially behind feature flag.

    • Partial read would reduce the IO.

    • Most of the time you may need to read full check to give it to

      bloac.

    • We are going to try to better understand the use case, and see

      how that could be percolated from the end user API to IO layer.

    • Mark unstable.

    • Andrew will do some of this.

  • Spec discussion of last week. Still keeping draft up to date: https://github.com/zarr-developers/zarr-specs/pull/71

    • How do you fallback ?

    • Dennis: interrogate tech spec about what types are available and

      let user decide fallback.

    • Alistair: only allow fallback on basetype.

    • Be restrictive in v3 and solve it in v3.1

    • Must_understand. Discussion:

      • What does HDf5 does.

      • Fail when you encounter.

      • Avoid locking the superblock.

        • You might not be able to lock.

    • Quidance in the spect with trailing / .

    • Maybe remove groups metadata documents.

    • Groups only have attributes.

    • Josh: possible workarounds

      • Duplicate metadata on the arrays…
      • Create an extension for the subresolutions…
      • Matthias: a differently name-spaced file?
    • Dennis: think flat is a bad idea. Would prefer not being able to

      get to an array without walking the groups. Containment leads to a simpler semantic.

    • Alistair: No support for parallel write of attributes of arrays.

    • Disagreement if underlying storage system have locks: seem to

      want to assume store does nto have locks.

    • John: (generally) keeping v2 users in mind. What drove adoption?

      • Alistair: starting from object storage since FS should comply
      • John: consolidated metadata – being able to get multiple keys in one request is an issue. Perhaps a question for API design.
      • Matthias: how willing to have API async / non-blocking?
      • Alistair: in Python? Yes. Good idea. See ongoing issue.
      • https://github.com/zarr-developers/zarr-python/issues/536

  • Draft of v2/v3 adapter: [DRAFT] V3 spec implmentation. · Issue #568 · zarr-developers/zarr-python

  • Closing: 21:40 UK