2023-03-16

Note: This ZEP meeting was an impromptu meeting. Please see the corresponding message on Gitter here.

Attending: Sanket Verma (SV), Norman Rzepka (NR), Jonathan Striebel (JS), Jeremy Maitin-Shepard (JMS)

TL;DR:

NR was confused about sharding as a codec and thought changing it would require rewriting the interface. JS and JMS answered NR’s questions. After this, we discussed sending ZEP1 for voting and concluded that there needs to be an editorial change before we send it. JS initiated a discussion on #161 and #222. NR also asked if we have a contributors list for V3, and SV took the job for himself.

Meeting Minutes:

  • NR: Confused for sharding as a codec! - Making sharding as a codec will change the interface -
    • JMS: doesn’t agree with Martin’s point
    • JS: doesn’t change the codec - will nest a new one - V3 doesn’t need to change anything
    • NR: Need to change ZEP2 then
    • NR: Zarr-Python API is mess right now

~JMS joins in~

  • NR: Should we send ZEP1 for voting?
    • JMS: Martin wants to push everything as a storage transformer - keep storage transformers in the spec - we could also defer that decision to ZEP2
    • NR: I wonder if we have too much implementation detail in V3? - Whether we need partial read or not?
    • JMS: Partial read are not required for sharding
  • JS: https://github.com/zarr-developers/zarr-specs/issues/161
    • JMS: Mostly concerned with JSON metadata - haven’t starting doing the implementation
    • NR: Some behaviour needs to be defined - everything goes beyond - doesn’t need to strip the spec
    • JS: hierarchy discovery - what happens if you delete the chunks and what happens then?
    • JS: We’re fine as it is now!
  • JMS: For someone is reviewing the ZEP0001 - JSON metadata is important - but it is burried in the middle - an editorial change would be helpful to put on the top
    • JS: Glossary defined at the top is not optimal
    • JMS: Would look into restructuring the metadata to the top
    • JMS: Would start working on V3 implementation of Neuroglancer and tensor store
  • JS: https://github.com/zarr-developers/zarr-specs/pull/222 -
    • NR: This is something of an implementation detail
    • JS: Maybe we can mark it them as implementation detail
    • JMS: Josh and Ryan brought up the idea of codec vs transformers in the last ZEP meeting - so I wrote this PR
  • NR: Move sharding to a codec?
    • JMS: Josh was skeptical - Ryan was in favour - we should go ahead with the proposal for sharding as a codec
    • NR: Will make the changes to the ZEP2 to the next week
    • JS: Having sharding as similar to blosc2 and nvcomp is a strongest opinion
  • NR: Contributors for V3 so far?
    • SV: Will make a contributor list after the voting email goes out!