9th September, 2020

Attending: John K, Martin D., Matthias B., Trevor Manzt, Josh Moore, Ryan Williams, Alistair Miles

  • Wheels:

    • Josh: everyone ok for it to move forward? Yeah. People were

      holding off just because of non-familiarity with wheels in general.

    • John: trying to get Czaki to maintain

  • Standing items

  • Core

    • Josh: mostly a heads up –

      https://github.com/saalfeldlab/n5/pull/73

    • Alistair: from last community meeting

      • https://github.com/zarr-developers/zarr-specs/issues/90

      • Q: what do you do about stores that can store metadata natively?

      • Josh: agree that there’s no conflict in v3, but felt there was in v2.

      • Alistair: not if you don’t have one hierarchy but a tree of separate documents

      • Matthias: not as much the storage as the API that you expose to users

      • Martin: lots of opportunities for this. Async of blobs is something that may not be in all implementations.

        • Does that get built into the core library?

        • Delegated to the storage layer?

        • Labeled as “experimental”

      • Matthias: for attributes, way to generate API in python that we couldn’t be done in compiled languages (i.e. knowing all attributes that exist)

      • Matthias: also, will such an API generate more traffic? (Transaction API)

    • Matthias: Keep Python 3.5 support until when ? Personally I feel

      the pain on supporting Python 3.5,

      • Some syntax (variable type annotation) is 3.6+ only, async/await keywords are 3.6+ only

      • and there a couple of other API, libraries/ libraries version that are only available on 3.6+.

      • Last “bugfix” python 3.5 was Python 3.5.4, Aug. 8, 2017 and later versions are “source-only security fixes”.

      • I understand that many people might be stuck using it. But as an open source project, there is a developper and contributor cost in keeping this, in particular fixing CI failures. For information* NEP 29* suggests to be 3.6+/ numpy 1.15+ around Jan 2020, 3.7+ around June 2020. Stable IPython is already 3.7+. And earlier version of Zarr will still work on 3.5….

      • John: https://devguide.python.org/#status-of-python-branches

        • Move to one last release bumping numcodecs

      • Josh: personally agree, think we need an issue proposing based on

      • Document “we support X versions” (and/or we drop when upstream says Y).

        • Matthias: moving forward there are going to be more

          releases so “# of versions” may shrink the range.

  • Matthias: If not much happening today can we use this time to review

    PRs ? There are 36 open PRs. In particular much of refactor and cleaning is in conflict with draft v3 spec impl, but would be useful to make things easier to work on / check, as well as do more refactor cleanup on top.

    • Non-draft test-success

    • Non-draft test-pending (most of those are “stuck”, can someone with rights on the repo push an empty commit or kick the tests?)

    • Non-draft test-failure (many of those are marked “WIP”, can maybe convert to draft ?

    • FSStore (Martin)

      • Josh: big thumbs up

        • Ah, remember that my tests went missing
        • Martin: +1, can add an existing test suite

      • Alistair: consolidated is handled in the store. Handle it

        above?

        • Martin: could and would be valid. Fits discussion on composable stores
        • This covers writable consolidated (only good for a single threaded)
        • Wasn’t sure.

    • Matthias: add close() to base class?

    • Alistair: nested-storage in FSStore?

      • Martin: For composability (nested+s3)

      • Josh: is this something

      • Alistair: think it would be nice to just have a

        transform-layer

      • What about detecting which chunky separator?

        • Josh: would love to have it!
      • Trevor: having composable stores sounds interesting. Have

        been wrapping various image types.

      • Martin: agreed that a transformer is beneficial, but still

        want the argument in FSStore for right now (since it’s been around for a while)

  • Tabled

    • Josh: (perhaps largely for the Pangeo crowd) anyone from OGC to talk to about the best way to store spatial indexes in Zarr? In higher than 2-dimensions? (Will bring up again next time)