15th June, 2022
Attending: Sanket Verma (SV), Ryan Abernathey (RA), John Kirkham (JK), Jackson Maxfield Brown (JMB), Dennis Heimbigner (DH), John A. Kirkham (JK), Martin Durrant (MD)
- ZEP acceptance criteria: https://zarr.dev/zeps/active/ZEP0000.html#how-does-a-zep-become-accepted
- GSoC 2022 coding period has officially started! Check the progress for Registry Codecs and Benchmarks.
MD: Weekly tracking of GSoC 2022 Kerchunk contributors here: https://github.com/fsspec/GSoC-kechunk-2022
- Non-zero origins: https://github.com/zarr-developers/zarr-specs/pull/144
- RA: JM’s proposal like a comments/suggestions to the ZEP1?
- JM: Yes, it’s like a comment but not a full suggestion
- JM: Having a non-zero origin as extension will be fine. Zarr doesn’t have a well defined coordiante space - if you add non-zero origin you need to add stuff when dealing with other types of arrays like reading or writing it to other file systems or arrays
- RA: uses Zarr also as a lower stack array - comfortable with the idea raw array space and coordinate space - and good with zarr doesn’t know about coordinate space - Xarray can build coordinate space - works with the metadata concept - Zarr can’t make Julia use index base - coordinate space is not suggestion, here we are changing the array index
- JM: certainly see Ryan’s argument - can use other libraries like Xarray for doing the index manipulation - also value of having array where you can talk about position
- DH: NetCDF coordinate system talks about latitude and longitude - introduce notion of coordinate variables - agree with Ryan’s - index level needs to be pure and standardised - whole variety of coordinate system that can be imposed later on - there are arbitrarily number of coordinate system that people use and bad to pick-up one here
- MD: agree with Ryan - in Xarray we can define coordinate system using other variables
- RA: JM also commented on the issue that the risk of not having in the core would be that client opening the Zarr arrays and would not able to access the array
- JM: unfortunate how Julia changes index - if you don’t talk about base index it doesn’t hurt anyone
- RA: HDF group is used to this - zero base indexing - language determines the exposition the array data -
Xarraycan do it because it has a data model in Xarray - diffuse this out in zarr - we have a primitive array storage system and on top of the we have various conventions of metadata and that’s the beauty - no explicit support is required for that - many tools can open that
- RA: We can put a convention to address the issue - a page of conventions on the website, something like https://zarr.dev/conventions can document that - processing softwares can use those - Zarr ontology to other array ontology - if we put it up in a Zarr core why are we catering to microscopy group only and why not the Geo community!?
- MD: The word
conventionis super useful - if you have tools which can leverage the indexing
- RA to JM: if we don’t support in the core array - it’s also about the implementation - have you thought about implementation?
- JM: very simple if
notin core spec - pretty clear boundary on how transformation can be done in dense integer space of Zarr array - index by coordinate array and other method - different data types where indexes are latitude longitude - having a extra level of translational array
- MD: Zarr array core design would need to behave like every language
- JM: if array is small - it’s in the memory and you can do a lot of stuff like read it store it and play around with it! - Zarr array and memory works in other ways!
- MD: naively do it in any language - use the language rules - you’d the do the selection as the array is in the memory
- JM: shifting the coordinate space - what about negative indexing? - How does Xarray handles it?
- MD: not possible - each variable has unique set of coordinate - the NetCDF conventions would not allow it - NetCDF conventions are far more rigid that anything - Xarray could certainly implement wide range of mapping - Xarray is born out of
- JK: negative indexing - logical indexing and coordinate indexing problem - data exists somewhere and how we map that to meaningful coordinate
- MD: negative indexing is problem in Python and means a different thing over there
- JK: big change - specifying changes by coordinate having to list them in metadata and to update the metadata for all the previous arrays
- MD: the reference file systems could do the renaming - but it is complicated
Discussion on: https://github.com/zarr-developers/zarr-specs/issues/141
- DH: is the issue representing the floating point numbers?
- JM: the attribute model is
.json- some way to intended as the number
.jsonhas that distinction
- JM: Python implements the extension - generally extension doesn’t support that - in JS you need to write your own parser to take care of this - no way to represent this and we need to discuss this
- DH: binary and nan will be represent as bit pattern
- JK: would love to see how coordinate space stack would look like - interesting to have it in extension - if Xarray would be interested in that - recasting the coordinate? - coordinate space extension? - changes the metadata? - graduate the metadata and see how to it behaves when the coordinate system is changed?
- JM: a few things that needs to be discussed:
- Zarr attributes and .json and infinity values binary things - wonder if there’s a solution to that in zarr v3 (https://github.com/zarr-developers/zarr-specs/issues/141)
- Zarr V2 array creation has a easy way to create arrays - whereas you need to mention path in V3; Zarr v3 array creation is pain because of path - could be handled by the having
//or any other special character to handle it