1st June, 2022

Attending: Davis Bennett (DV), Sanket Verma (SV), John A. Kirkham (JK), Trevor Manz (TM), Brianna Pagán (BP), Parth Tripathi (PT), Gregory Lee (GL), Ward Fisher (WF)

Updates (SV):

  • Release 2.12 - Blog and Tweet
  • ZEP website: https://zarr.dev/zeps
  • ZEP 1 - Alistair and Jonathon working on it, check here. PR here
  • David(xtensor), Ward(NetCDF-C & Java), Trevor(zarr.js) and Gregory Lee(zarr-python) added to the ZIC
  • GSoC 2022 contributors updates - Shivank and Parth

Agenda:

  • TM: https://excalidraw.com/ for making graphic representations, TM used them for the Zarr slides 🖼
  • fill_value for empty chunks:
    • DV: How could it implemented to avoid breaking changes?
    • DV: Whether there is a clear description about fill_value in Zarr Spec V2?
    • JK: V3 spec aims to mitigate the issue of fill_value and have clear text about it
    • DV: Haven’t looked at V3 but if it specifices fill_value by default then it’s good
    • TM: Not writing fill_value is sematically correct, on the implementations side: having to implement something like this is more flexible and easy in JS as compared to Python
    • DV: Problems representing numbers as .json metadata. There should not be constraints to fill_data/fill_value in .json as it’s metadata
    • TM: Serialising the scalar arrays as containers
    • JK: Slightly hard to read metadata with fill_value as there are amiguity issues
    • JK: How does NetCDF handles fill_value?
    • WF: Couple of different approaches to handle it.
      • We do it for standard and compound data types
      • In absence of user-input we have a default fill_value
      • Mostly users need to specify the data type as it’s difficult to come up with it on it’s own
      • There’s also a flag tp supress fill_value
      • As NetCDF is mostly used for data archival and for that we need to maintain data integrity, it’s difficult to know what’s done if there are no defined data types or no fill_values
      • We can also explicitly write fill_value
      • Store in metadata
      • JK: Is there a concept of not storing it?
      • WF: fill_value can be changed, the default behaviour is to replace old fill_value to new fill_value, not the best case but yeah, it is what it is!
  • TM: FSSPEC reference ability to record base64 to .json. How data URLS work: https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs
  • SV: Conferences 📣
    • BP: Speaking at ESIP(in-person) on July 19th-22nd on cloud processing and how they process internal Zarr & COGs data at NASA. Christine will be talking about Zarr and almost 50% of the content will be focused on Zarr. Link: 1 and 2
    • WF: Will be talking about NCZarr in Rocky Mountain(hybrid), current state of NetCDF. Link to follow in the Gitter chat.
    • JK: GTC(virtual) happened in spring, NVIDIA is presenting a few things at SciPy
    • SV: CFP for EuroSciPy is open until 6/6 AoE, submit here
  • Tabled
    • ZEP1