18th December, 2019

Attending: Ward, RyanW, Josh, RyanA

• Standing items

  • Introductions: new joinees should feel welcome to introduce

    themselves and their interest in zarr/n5.

• AG(eophysical)U [Ward]

  • Poster on Zarr/NetCDF well-received. Got a crowd.

  • Hoping to have the first version out in the next month or two.

  • Team from NASA had a lot of questions for a new mission. E.g.

    won’t need to make a decision between Zarr & NetCDF.

  • Dennis has been doing great work. Hopefully something to test

    soon and we can start finding corner cases.

  • High-level take away: lots of excitement at AGU.

• Other file formats – talking points

  • Options: merging, diverging, …

  • Institutional structures, community support, funding

    opportunities

  • Decoupling data models & data storage

  • Strength of spec-first design philosophy

  • Existence of third-party implementations

  • Cf. Matt’s [*dask comments about

    spark*](https://docs.dask.org/en/latest/spark.html) (fair, balanced, etc)

  • Sharing our stories

  • See also:

    https://github.com/Blosc/Caterva

• Ryan’s performance benchmarks (very preliminary):

  • https://github.com/rabernat/pangeo-throughput-benchmarks/blob/master/plot_results.ipynb

  • Prep for a paper

  • Include disks & HPC

  • Worst performance is with very small reads. Trade-off in the

    (cloud) environment.

• CMIP6 data in the cloud:

  • https://medium.com/pangeo/cmip6-in-the-cloud-five-ways-96b177abe396

  • https://cloud.google.com/blog/products/data-analytics/new-climate-model-data-now-google-public-datasets

• NumFOCUS

  • RyanW to send the blurb. Josh can step in as needed.

  • RyanW will approach J.B. for contracting.

• Proposed next date: January 8, 2020