2023-05-31

Attending: Davis Bennett (DB), Josh Moore (JM), Jeremy Maitin-Shepard (JMS), Norman Rzepka (NR), Brianna Pagán (BP)

Apologies: Sanket Verma (travelling)

TL;DR:

DB started the discussion by pointing out a conceptual hole in Zarr-Python. BP gave an update on Geo-Zarr spec - the first draft will be submitted to OGC this week. DB had some questions on Geo-Zarr, which BP took care of. JM shared his thoughts on community building and ZarrCon.

Meeting Minutes:

  • DB: conceptual hole in zarr-python
    • array constructor doesn’t get called, it’s init_array
    • there’s no representation of an array that’s abstracted from storage
    • if you want to define a heirarchy and not store it, you have no option
    • JM: zarrita? NR: lost the hierarchy class and depends on a store. memory store?
    • DB: pydantic_zarr walk-through
    • DB: (tl;dr) don’t think we want pydantic as a dependency of zarr-python, but could be applied on top for validation
    • NR: also didn’t want the dependency in zarrita
    • JMS: 👍🏻 for the representation detached for the storage. currently no json-schema for the v3 spec, but could use the dataclasses for that (also to generate language bindings)
    • DB: typescript has a lot of generation; but not in python. does typescript have more money behind it?
      • JMS: just better designed perhaps
    • JMS: sphinx extension for documenting json structures
  • BP: GeoZarr SWG OGC draft submission this week
    • https://github.com/zarr-developers/geozarr-spec/pull/23
    • JM: :tada: Anything that needs discussing or just FYI?
    • BP: Just an FYI, although some questions on where pyramiding is in terms of a ZEP. The relationship between some of the geospatial needs that are agnostic to geospatial community, we don’t want to specify in GeoZarr if it’s work needed on a ZEP. Pyramiding is one of those pieces.
    • JM: https://github.com/zarr-developers/zarr-specs/pull/50 was the original attempt to propose multiscales for/in the zarr community but it was “kicked out” (:smile:) to be an external convention.
    • BP: definitely work we would be interested in and maybe we can dedicate some dev time to pick up wherever this left off? I wanted to touch base with: https://github.com/carbonplan/ndpyramid, but feel a bit lost - not sure if carbon plan then is submitting an external convention?
    • JM: I don’t know if they plan on proposing something. They were using the OME-Zarr/NGFF multiscales in the early days, but are much closer to the GeoZarr community. Happy to discuss (e.g., extracting out a common representation)
    • BP: ❤️ can we schedule a coffee chat?
  • DB: difference between geozarr and serialization of xarray into zarr.
    • BP: naming of attributes, etc.
    • DB: closer to CF conventions?
    • BP: yes. Basically adopting it. But now looking at limitations. Bringing in people from Geo-TIFF community. Going through OGC SWG process. Trying to figure out what the scope is. multiscale is upstream.
  • JM: this fits well with the previous model/pydantic discussion.
    • JM: but not a great place for it to live at the moment
    • JMS: haven’t worked out how conventions would work. perhaps keep it to discrete coordinates (OME is continuous coordinate spaces)
    • BP: advertising GeoZarr as waiting for ZEP4
    • JM: time pressures? BP: making some zarr stores available, but before we generate them, it would be good to have some consensus. “zarr stores by the end of the year” (with OGC stamp of approval)
    • BP: how to move forward? JM: someone to pick Ryan’s brain about https://github.com/zarr-developers/zeps/pull/28
  • DB: Geo-TIFF / Geo-Zar / ?!
  • JM: slight sidenote to the difficulty of bridging communities which gets harder the more people you bring to the table
  • JM: ZarrCon? BP: ESIP based. That was the deadline for the SWG. Wanted to present it and ask for stakeholders. Hosting an xarray / zarr session. (And another cloud native session). AGU / EGU? Proposals just closed for AGU…