8th September, 2021

Attendees: Josh, Dennis, Hailey, Eric, Greg, John Kirkham, Davis

  • NetCDF

    • Hailey: netcdf-java codec framework set up, but haven’t chosen

      the implementations.

      • Using a ServiceProvider implementation (answering to ID or Name)

      • Codecs can be added by users (usable, but not built in)

      • Otherwise, read-only & AWS S3 SDK.

        • Eric: abstraction level!

        • Josh: at least in Java,

          FileSystem.

        • Dennis: priority? Eric: S3 > GCS > Azure

        • API is ok, but authorization is hard. Otherwise could

          build own.

      • Reduced capacity moving forward.

    • Dennis: PR for supporting filters in Zarr (merged to master)

      • Looking for a S3 client library. Old version of AWS SDK C++ works. Newer dies. AWS Labs C dies (different reason)
      • License needed: non-GPL.

  • Josh: point clouds (e.g. potree)

    • Any connections?

    • Dennis: similar (big) issue in NetCDF. Whole subsystem in Java.

    • Hailey: might have pulled it out.

  • Dennis: status of N5?

    • Josh: (report)

    • Dennis: could netcdf-java make use of the filters

      implementation?

    • Eric: issues with jzarr (doesn’t work with blosc on mac)

    • Josh: could collaborate on the codec interface. don’t think

      anyone has proposed anything.

    • Josh:

      https://github.com/glencoesoftware/bioformats2raw/pull/89 for example of using FileSystem

  • Davis: empty chunks

    • Dennis: expensive?

  • Davis: any chance of seeing sharded format

    • Josh: report on webknossos. Need a C API.

    • Will that bubble up into the Zarr API?

    • Davis: violates current write in parallel simplicity

    • Eric: append only sharding. or read-only? Then efficient access

      suffices.

    • Davis: you can also use space-filling curves (morton, etc.)

      • Optimization of range requests

    • Eric: superchunk & chard are interchangeable

    • Davis: don’t like the “shard” terminology.

    • Dennis: variable size chunking? Eric: adaptive meshes could use

      it, yes. (but lots of complexity)

      • Eric: possible currently with dask lazy arrays
      • Dennis: perhaps meant change chunk size and have it overtime migrate.
      • Synchronization issue