2023-11-01
Attending: Sanket Verma (SV), Josh Moore (JM), Jeremy Maitin-Shepard (JMS), Dennis Heimbigner (DH)
TL;DR:
DH discussed implementation details of V3 specification for NetCDF. He has been working actively on his implementation these days. JM gave him the idea of proposing a convention on NetCDF’s V3 implementation for the community. JMS thinks we should have a compatibility matrix on our website for users’ reference. Also, we discussed the pain of daylight savings time. 🥲
Updates:
- ZEP0002 Voting - No vetos! 🎉
- Greg will be voting soon - Greg’s comment
- Zarr-Python updates
- Lot of new PRs merged recently - thanks Davis, David, Dmitri, Ziwen and Josh and others! - maybe a new release soon?
- B&P group meeting
- Jack used Intel VTune to profile the read operation of Zarr-array with different sizes - https://github.com/zarr-developers/zarr-benchmark/discussions/22
- Meeting notes: https://docs.google.com/document/d/1s5VBWqyh_MliVZCy9KK8ZnMy2pIeKF5pTLI2XWZ0MNE/edit?usp=sharing
- Refactor group meeting
- Creating a a new V3 namespace within Zarr-Python to isolate the development of V3 from existing code
- Notes: https://hackmd.io/Rbh6oae8S7mNU-CPWWwPaw
- Publishing notes on the website soon!
- Any updates from the community?
Meeting Minutes:
- The discussion on daylight savings time
- DH: Discussion on V3 implementation - Zarr-spec issues
- DH: Needs to work with inferencing in order to make V3 work in NetCDF - it’s doable - but there might be consequences
- JM: can layer on absolute/graph-like semantics later? DH: yes.
- DH: Worry - user reading the metadata might get confused or make an unwanted assumption
- JMS: NetCDF model has scoping to them? DH: Yes!
- DH: Dimensions can be declared as objects in the group
- DH: what were named dimensions for?? (look like macros)
- JMS: seem to be multiple visions of what named dimensions are for. referencing by name in xarray as opposed to index
- JMS: neuroglancer assumes based on the name (interactive tool)
- DH: can’t assume anything based on the name. Nothing in the spec, to prevent different sizes for the same dimension name (Not the case in NetCDF community that that’s an assumption people want to make. esp. with cfconventions.)
- DH: zarr doesn’t need groups. nothing more than a namespace. could encode that in the array name.
- JM: except group attributes.
- JMS: used by higher-level spec.
- JMS: matches the original theory of v2. True that v3 originally had something more.
- DH: Making arrays self-contained
- DH: Adding bunch of metadata and inferencing would get me where I want to be - completing the V3 implementation
- DH: Need to make inferencing about types
- DH: Would also need to give user some ability for inferencing - but don’t know how to do it now!
- DH: Support for string data types
- SV: Currently a ZEP is in progress: https://zeps--47.org.readthedocs.build/en/47/draft/ZEP0007.html
- DH: What’s the largest length of the string?
- JMS: Currently, we want to add the support and the size is variable - nothing as fixed length strings
- DH: Would like to get fixed length string get in
- JM: Encode the attributes and present it as a convention that others will use
- JMS: Built a website for Zarr-Specs like Mozilla/Caniuse because the specs is pretty stable as of now
- JM: Could see building site for various tests
- JMS: Link implementations directly from the spec
- JM: Good to have a back channel to report if something is not working/faulty
- Examples: