26th January, 2022

Attending: Josh Moore, Ryan Abernathey, Eric Perlman, Hanka Medová, Sanket Verma, John Kirkham, Ward Fisher, Dennis Heimbigner, Greg Lee, Matt McCormick, Fabian Gans, Jackson Maxfield Brown

• Introductions

• Topics

  • C/F ordering

    • https://github.com/zarr-developers/community/issues/41

    • C-ordering is the default (natural in Python)

    • in Julia, typically looks reverted (in metadata) but without transposing it.

      • Seems to what most people who are doing it are used to

        this.

      • But there is confusion if you save in Python (10 rows, 5

        cols)

      • when in Julia it’s different.

    • Ryan: is it important to be able to write F-ordered data?

      • What’s important is what bits are close together not the

        language convention. May not have understood the issue.

      • Didn’t transpose but only change the metadata. Then can

        read as is.

    • Dennis: fortran impl. reverses the dimensions before it calls the C library which gives the correct order for the data. netcdf-c stores everything (by default) as row-major order. For nczarr, must look at the ordering and then decide whether or not to reverse.

    • Ryan: the fact that it’s in the spec but not supported is a problem.

      • Don’t see the use case for it right now.

    • ….lots of discussion…

    • Fabian: remember the reason for F-ordering is because compression in some orders is more efficient

    • https://github.com/zarr-developers/zarr-python/issues/7

    • John: not sure that’s (still) the case, since carefully try to never transpose.

    • Dennis: should be talking about column- and row-major ordering rather than C/F.

    • John: need to also discuss with N5

  • Xtensor-zarr implementation status (Matt)

    • Being applied here: https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF

      • API / testing issues in xtensor-zarr

        • https://github.com/xtensor-stack/xtensor-zarr/issues/58
        • https://github.com/xtensor-stack/xtensor-zarr/issues/57

    • Matt: running into some bugs. Working with team in France.

    • Josh: did you try tensorstore or z5? c++ netcdf is also an option.

    • C++ implementations:

      • https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF/issues/3

      • e.g. also

        https://github.com/usnistgov/FastLoader

    • Josh: would love to find a way to share code (Constantin could see moving z5py onto xtensor-zarr)

  • NetCDF WebAssembly support (Matt)

    • https://github.com/Unidata/netcdf-c/pull/2129, “Support hdf5debug.c compilation with Emscripten“

      • Todo: follow-up on netcdf-c repository GitHub

        Discussions

    • https://github.com/InsightSoftwareConsortium/itk-wasm

    • C#, etc. https://github.com/bytecodealliance/wasmtime-dotnet, https://wasmer.io/

    • Dennis: interested in using WebAssembly (was thinking about using it for compressors for Java). Perhaps move to discussion section, https://github.com/Unidata/netcdf-c/discussions

  • Xarray/Zarr (Josh & Jackson, minimally)

    • BOpen consultants working on:

      • NetCDF / xarray / Zarr compatibility

      • Hierarchical support

        • Data trees
        • Multiscale conventions in zarr repo:
        • https://github.com/zarr-developers/zarr-specs/issues/125

    • Jackson: aicsimageio only supports xarray for different locations (scenes, position or OME:images. some other dimension)

      • then multiscales, i.e. pyramids

      • datatree stuff should help. we have different groups and

        they MAY have multiscales.

      • seems like

        https://github.com/pydata/xarray/issues/4118 would work

      • don’t know if it’s “enough” for aicsimageio. (Jackon’s)

        dream would be to have multiple datatrees where each is a position.

      • Datatree

        Dataset (diff position / scene / image)
        Resolution 0
        Resolution 1_2D …
        Resolution 1_3D …
        Dataset (diff position…)

Resolution 1

• Fabian: would be nice to have multiple chunkings of the same

  data (Also from Brockmann Group in their viewer)

• Jackson: currently do this (fake it) with TIFF, CZI, LIF by allowing

  the user to say how they want to chunk (zyx, tzyx, …)

• (old reminder)

  https://github.com/thewtex/spatial-image-multiscale

• Time permitting

  • Meeting slot: poll to be opened.