19th June, 2019

Time: 20:00 BST (15:00 EDT).

Joining instructions see above.

Intend to attend: Josh, RyanW, …

Attending: Josh, Martin, Ward, John, Ryan, Tony

Standing items

Introductions: new joinees should feel welcome to introduce themselves and their interest in zarr/n5.

Items

• Ward: Unidata blog post.

  • https://www.unidata.ucar.edu/blogs/news/entry/netcdf-and-native-cloud-storage

  • Pretty good resonance

  • More technical details coming under developer blog

• Josh: upcoming bioimaging blog post as well

  • “No new formats”

  • With technical follow up as well.

• Ward: question

  • Meeting with various groups (incl. hdfgroup)

  • Re: HPC data storage

  • “How well does zarr perform in a super computer environment?”

  • Josh: Saalfeld’s spark usage at Janelia

  • Martin: may not be that much benefit on the reading side

  • Ryan: the question is do you have multiple write heads

• Benchmarks in general

  • Ward: generalizable dataset of benchmarks (existing or created)

    would be good

  • Dimensions: Chunk size, different filesystems (cloud, parallel,

    hadoop, local)

• Ryan: peripheral feature - sparse arrays

  • https://github.com/daletovar/zsparse

    (thin wrapper)

  • 3 1-d array is stored like a chunk zarr array

  • A few fixes sitting locally

  • Will be useful for single-cell work

  • Spec changes? No. Application level thing that interacts with

    underlying arrays.

  • E.g. wasteful to materialize all the nulls/zeros

  • Using numpy methods. One of them is the three 1-d arrays.

    Transforming indices from higher-dimensional space. You can pretend it’s dense.

  • https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format)

  • Got to this via an attempt to add a database backend. Want a

    virtual layer on top of zarr.

  • https://github.com/zarr-developers/zarr/issues/438

• NetZDF anything to test yet?

  • Ward: not yet. Will publicize that widely.

  • May be made such that it can be extracted.

  • Asked by community and leadership to be more vocal

    • Heard from Director. “Please put up a blog post”

  • Lots of interest

  • Contact? See generic link in blog post. (netcdf-support or ward

    directly. Or GH page. Upcoming zarr support issue from Ward.)

• SciPy (John)

  • Ward: Unidata (another Ryan May) will be there from Pangeo. One

    of the big Zarr boosters.

  • Logo to be spread:

    https://github.com/zarr-developers/zarr/issues/300#issuecomment-427064151


  • <img src=”Pictures/10000201000001D6000000F6A360F8EF8A47245F.png”

    style=”width:1.4953in;height:0.7819in” />

  • Start using; likely be another iteration though (post scipy)