Authors: Ryan Abernathey, Alistair Miles, Josh Moore, Norman Rzepka

In the ten years since its inception, Zarr has transformed the landscape of scientific data management. What started as a side project for a single genomics researcher frustrated with existing storage options has blossomed into critical scientific infrastructure for bioinformatics, bioimaging, geospatial, and Earth-system science. Zarr is in use in production at organizations such as NASA, NOAA, ESA, EMBL-EBI, Allen Institutes, Bi[o]hub, RIKEN, NVIDIA, Google, and Microsoft. Zarr was recently chosen as the next-generation format for the Copernicus Sentinel missions, which produce 40 Petabytes of data per year.

The reasons for this widespread adoption fall into two broad categories:

• Technical - Zarr’s design offers superior performance, scalability, and extensibility than alternatives, especially in the cloud. The data model can accommodate a wide range of domain-specific scientific data schemas.
• Social - Compared to data formats controlled by a single vendor, Zarr’s open-source, community-based governance model appeals to organizations who care about long-term data sovereignty and stewardship.

The social reasons are not to be discounted; for many organizations, these are just as important as the technical ones. So as the Zarr community evolves and grows, so must its governance. Learning, evolving, and growing is essential to any well-functioning organization, and we have learned many lessons from the past few years. Today we are proposing updates to Zarr’s governance aimed at streamlining the development process and encouraging new contributors to get involved while recognizing the importance of maintaining stability and continuity.

Current Challenges

Zarr governance started as a “BDFL model”; the creator of Zarr, Alistair Miles, wrote the initial spec, built the first implementation (zarr-python), and was the sole owner of these things. Recognizing the value of empowering the community, in 2020 the Zarr Steering Council and current governance framework was established while moving the project under the umbrella of NumFocus as a fiscal sponsor. During this period, a significant number of new Zarr implementations emerged.

Building and maintaining a data format in a decentralized and community-driven way, without one organization or individual ultimately calling the shots, is not without its challenges. Today the Zarr community is feeling growing pains around governance in several ways:

• Development of extensions and conventions has been somewhat slow moving due to an ambiguous approval process
• The agency of some individual projects within the zarr-developers GitHub org (e.g. zarr-python) is constrained by permissions limitations.
• Ambiguity around the process for membership on Zarr’s various committees and groups.
• Fragmentation of Zarr-related projects across many different GitHub organizations and repositories.

Our diagnosis of the root cause of these challenges is that Zarr governance has not fully evolved its single-project, single-repo structure to the multi-project reality of today. Additionally, project administrative issues have become intermingled with the constraints of GitHub’s IAM system, conflating governance with technical capabilities of this specific platform.

Going forward, our new proposed framework aims to clarify the roles and responsibilities of different stakeholders via more precise definitions.

What is Zarr?

The Zarr ecosystem today consists of the following entities.

• The Zarr Project - The umbrella entity which is fiscally sponsored by NumFocus. NumFocus sponsored projects are required to have a formal governance process and may receive financial donations via the NumFocus 501c3.
• The Zarr Specification - A collection of documents defining the on-disk Zarr format.
• Affiliated Software Projects - Specific individual software projects (including the specification but typically Zarr implementations) which are considered part of the Zarr Project. These projects are eligible to receive funds via NumFocus and are therefore subject to the overarching Project governance framework.
• Non-Affiliated Software Projects - Other software projects which implement Zarr or interact with it in some way, but are not under the umbrella of the Zarr Project.

Our current governance challenges mostly stem from incomplete disambiguation between these different entities, and the resulting overloading of the authority of the ZSC. This ambiguity is exemplified by the current governance doc, which lives at the zarr-developers org level. It’s unclear whether this applies to all repos within the zarr-developers org, and whether it applies to zarr-affiliated projects under other orgs. We aim to resolve this through the following resolutions.

1. The Zarr Project as a whole will remain governed by the Zarr Steering Council ZSC. Going forward, the Steering Council’s responsibilities include:
   1. Interface between The Project and its fiscal sponsor
   2. Manage the copyrights and trademarks associated with the Project
   3. Manage the list of affiliated projects
   4. Manage responsibilities which don’t belong to a specific affiliated project (e.g. Zarr website, GitHub org) in order to ensure smooth operations and effective collaboration.
2. The Zarr Specification will be governed by a separate Spec Committee. The Spec Committee’s responsibilities are to
   1. Manage changes to the Zarr core Specification
   2. Maintain the list of Zarr Extensions
3. Each Affiliated Software Project will be governed independently by a Core Developers Group (CDG) and will adhere to a simple governance process defined in a standardized document (below for more detail), without oversight from the ZSC.
4. We are introducing a simple and transparent process for welcoming new Affiliated Software Projects.
5. Non-affiliated Software Projects can of course continue to do their own thing however they wish, outside the boundaries of this framework, with whatever governance (or lack thereof) they choose. Non-affiliated projects are not eligible for direct funding via NumFocus. We welcome and encourage all Zarr-related projects to become affiliated.

These changes are formalized as a Zarr Enhancement Proposal (ZEP-11), which will now go through an approval process and receive feedback from the community. Feedback should be shared via the dedicated discussion issue in the zeps repo.

The Spec Committee

The Zarr Spec is the focal point that brings together all of the different Zarr implementations. Because it addresses the on-disk format, decisions about the spec will persist for decades. Ensuring responsible and careful evolution of the spec, and balancing the need for innovation with the need for stability, is the challenge of the Spec Committee.

The evolution of the spec is currently governed by the ZEP process (see ZEP0). Changes to the spec currently require unanimous approval of the ZSC and majority approval of the Zarr Implementation Council (ZIC). In practice, the ZIC didn’t take shape in the way we had hoped. As implementations appeared and became inactive, the council’s membership adapted too slowly. Members frequently didn’t have the time needed to follow the ongoing GitHub conversations. The steering council also never set expectations around meetings or time limits on when decisions would be made. Looking back, the lack of clear time commitments and processes made it difficult for the ZIC to function effectively.

Instead, we propose moving to a single Spec Committee. This is what other successful, widely adopted file formats (like Apache Parquet) do. To provide continuity with the current system, initial membership of the Spec Committee will include the current members of both the ZSC and the ZIC; however, these members are encouraged to resign if they don’t plan to actively participate. It is expected that the Spec Committee will contain members of the most active Zarr implementations, including both Affiliated and Non-affiliated Software Projects.

The Spec Committee’s primary mandate is to manage the core specification and to review and approve Zarr Extensions. To streamline decision-making and ensure agility, the first task of the newly formed Spec Committee will be to establish and document its own operational governance model, including decision-making procedures (e.g., voting mechanisms and quorum requirements) and membership review. This new process will supersede the unanimous ZSC/majority ZIC approval process currently defined in ZEP0 for all future core specification changes.

Governance for Affiliated Software Projects

Affiliated Software Projects will now follow a simple, standard, merit-based governance process akin to the Apache model. The default governance for Affiliated Software Projects is spelled out in an accompanying governance document: Draft: Governance for Zarr Affiliated Software Projects.

This governance initially consists of the following key elements:

• Each project has a group called the Core Developers Group (CDG) which makes decisions, e.g. about accepting PRs. (A group of one is fine for small projects.)
• Projects aim for consensus, falling back on majority vote of Core Developers when necessary.
• Any contributor is eligible to join the CDG. Existing Core Developers can nominate new members. Nominations should be based on evidence of sustained, quality contribution to the project. Nominations are accepted by majority vote of existing Core Developers.
• Core Developers who become inactive can and should be removed, via a majority vote.
• Larger groups should have a chair, who acts as a facilitator / coordinator.
• Projects must adhere to the Zarr code of conduct, which is a requirement for NumFocus fiscal sponsorship.

Projects are free to evolve and change their governance as they see fit, provided it remains within the accepted norms of community open-source projects. If an affiliated project abandons open and transparent governance, the ZSC reserves the right to remove its affiliation.

Current affiliated projects are:

• The Zarr Specification
• Zarr-Python (incl. Numcodecs)
• Zarr-java
• jzarr
• GeoZarr
• VirtualiZarr

Going forward, we welcome new affiliated projects to join. To be considered for affiliation, a project must be open source, be directly related to Zarr, and demonstrate a critical mass of sustained development activity. The ZSC will make decisions about affiliation based on these criteria and the overall strategic direction of The Project.

Non-Affiliated Software Projects

The Zarr ecosystem contains several important implementations that are not formally affiliated with the numfocus-sponsored Zarr Project. These include Tensorstore, Zarrs, and Zarr.jl. This is great and speaks to the vitality of the Zarr ecosystem.

For projects which do wish to be formally affiliated with the Zarr Project, and thereby benefit from the NumFocus fiscal sponsorship, we extend an open invitation to join. We hope the governance changes introduced here make this appealing and clarify the light-weight governance expectations around affiliated projects. Conversely, non-affiliated projects are totally fine as-is and are under zero pressure to change their approach to governance.

The Role of GitHub

The use of GitHub by the Zarr Project is purely an implementation detail. GitHub is a tool which helps us manage code. GitHub roles, groups, and repos do not have any formal status within the Zarr governance framework.

That said, in practice, we are heavily reliant on GitHub for day-to-day operations, and clarity is needed about how to map various roles and responsibilities to specific capabilities and functions within GitHub. Zarr is mission critical code for hundreds of organizations, and security must be taken seriously. Because privileges in GitHub map to ability to commit and release code, we aim to adhere to a “principle of least privilege” model, wherein different actors have only the minimal privileges required to perform their function. This minimizes the blast radius for any potential threat. For example, a compromised GitHub account for a maintainer in one affiliated project should not be able to impact a different affiliated project.
Consistent with existing practice, we make the choice that all affiliated software projects will live within the same GitHub organization: zarr-developers. The reasons for this are:

• It provides a central entry point into all outputs of of the NumFocus-sponsored Zarr Project, aiding discoverability
• It allows affiliated projects to share resources, including paid capabilities of the GitHub platform

Each Affiliated Software Project, including the Zarr Spec itself, will have responsibility for one or more repos under this parent organization. Each affiliated project will manage its own committee membership via a GitHub group. Ownership of the GitHub org as a whole will be managed by the ZSC, including creation of new repos and the creation of groups.

There is one technical limitation of GitHub which must be addressed: new members to the organization can only be added to existing groups by organization owners. This makes the ZSC a bottleneck on the autonomous operation of each affiliated project. As a workaround, we will set up a github bot which automatically updates group membership based on a list of committee members within each repo. This will effectively allow each affiliated project to manage its own membership independently.

What’s Next?

It’s an exciting time for Zarr. The recent creation of the extensions and conventions frameworks are motivating lots of creative new directions for development. Our goal with these governance updates is to cast off the last vestiges of Zarr’s early governance and replace it with a model that is appropriate to the multifaceted character of the Zarr Project today. This model is based on proven, established best practices for multi-stakeholder, community-developed open-source software projects. Once fully implemented, these governance updates will empower developers to work more effectively and eliminate unnecessary centralization of authority within the ZSC.

Zarr-Python 3 is available today on PyPI and Conda-Forge. It is compatible with Python 3.11 and above.

pip install --upgrade zarr
# or
conda install --channel conda-forge zarr

Support for Zarr’s v3 specification

The most notable addition in Zarr-Python 3 is complete support for Zarr’s v3 specification. The v3 specification brought greater multi-language interoperability and new extension points for customizing Zarr (codecs, chunk grids, data types, and stores).

Beyond supporting the core v3 specification, Zarr-Python 3 also includes support for the chunk-sharding extension. This feature allows for multiple chunks to be stored in a single file (or object), allowing users to utilize much smaller chunks without increasing the total number of objects in a dataset. Without chunk sharding, users optimizing for read-heavy applications had a difficult choice: either use a small chunk size, but create a huge number of stored objects, or use a large chunk size, but suffer poor IO for random reads into the data. With chunk-sharding, the number of stored objects is decoupled from the chunk size. Users can safely create very large Zarr arrays with very small chunks without generating a glut of stored objects. For more on how sharding works, see Zarr-Python’s sharding documentation page.

The code block below show’s off Zarr-Python’s new API for creating sharded arrays:

import numpy as np
import zarr

arr = zarr.create_array(
    "data/example-1.zarr",
    dtype="int32",
    zarr_format=3,
    shape=(1000, 1000),
    shards=(100, 100),
    chunks=(10, 10),
)
arr[:] = np.random.randint(0, 100, size=(1000, 1000))

Note that Zarr-Python 3 maintains read and write support for data stored according to Zarr’s v2 specification. Some features (e.g. sharding) are not available for v2 data. Users can set zarr_format=2 in the top level API to continue using Zarr v2’s specification.

Major performance improvements

Zarr-Python 3 delivers significant performance improvements across the board. A large part of the refactor focused on making the core of the library fully asynchronous, using Python’s asyncio library. The new asynchronous core enables efficient I/O operations and better utilization of system resources. This means that multiple I/O operations can be performed concurrently, leading to faster data access and reduced latency, especially when data is stored on high-latency storage backends (like cloud object storage).

For compute bound operations (like compression/decompression), Zarr now dispatches to a managed thread pool. Combined with asynchronous IO, this threaded parallelization allows for Zarr to take full advantage of the compute resources available when reading and writing data.

While early benchmark results appear to show very promising performance results relative to prior versions of Zarr-Python, we have yet to do dedicated performance tuning. Users should expect further performance improvements as Zarr-Python 3 matures. In fact, we are already working on identifying and addressing a number of known performance bottlenecks to further enhance the library’s speed and efficiency.

Built with extensions in mind

Zarr-Python 3 is designed to be highly extensible. Key features include:

• New Store ABC: A new abstract base class for defining custom storage backends, making it easier to integrate Zarr with various storage systems. This allows for seamless integration with cloud storage solutions, distributed file systems, and other data storage technologies.

  Zarr-Python 3 includes for the following stores:

  • LocalStore - for reading/writing to a local file system
  • FsspecStore - for reading/writing to remote/cloud storage (based on fsspec)
  • ZipStore - for reading/writing to a ZipFile (experimental)

  Additional stores are also in development (like Earthmover’s Icechunk store).

• Codec and CodecPipeline Entrypoints: Zarr-Python 3 provides Python entry points for defining custom codecs and codec pipelines, enabling flexible data compression and encoding strategies. This empowers developers to tailor data compression and encoding to specific use cases and optimize storage and performance.

  Numcodecs has been adapted to use Zarr’s Codec entrypoint system and Zarrs-python has already developed an experimental Rust-based CodecPipeline.

Modernized Codebase

The Zarr-Python 3 codebase has been significantly modernized:

• 100% Type Hint Coverage: Comprehensive type hints improve code readability, maintainability, and IDE support. This makes the code easier to understand, debug, and refactor, leading to higher code quality and reduced development time.
• Cleanly Defined Public/Private API: A clear distinction between public and private APIs enhances code organization and stability. This ensures that the public API remains stable and consistent, while allowing for flexibility and future development in the private API.
• Improved Development Environment, CI/CD, and Testing: A streamlined development workflow, robust CI/CD pipelines, and comprehensive testing ensure high-quality releases. This rigorous development process helps to identify and fix bugs early, leading to more reliable and robust software.

Migration from Zarr-Python 2 to 3

We have done everything possible to make the migration from Zarr-Python 2 to 3 as easy as possible. The 3.0 migration guide provides details the parts of the Zarr-Python API that have changed and provides suggested actions for migration. Additionally, libraries such as Xarray, Dask, have already added support for Zarr-Python 3.

Conclusion

Zarr-Python 3.0.0 marks the beginning of a new chapter for the Zarr-Python project. We encourage you to try out this new version and provide feedback. We’re also excited to see the development of new extensions built on top of this solid foundation, such as Icechunk, Zarrs-Python, and VirtualiZarr.

The development of Zarr-Python 3 was a huge effort, spanning over 12 months and including contributions from over 30 contributors. Special thanks to Davis Bennett and Norman Rzepka who helped me kick off the initial refactor in Potsdam, Germany in December 2023.

Further reading

• Zarr-Python 3 documentation
• Zarr-Python 3 design doc
• Zarr v3 specification

~Joe Hamman

This post will explain what our old versioning policy was, why it wasn’t working well for us, and why we are switching to “Intended Effort Versioning”, or “EffVer”.

Our old versioning policy

The old Zarr-Python versioning policy was effectively Semantic Versioning, or “SemVer”. In this scheme, backwards-incompatible API changes may only be released in major versions, backwards-compatible API changes (i.e., adding new APIs) may only be released in minor or major versions, and bug fixes can be released in major, minor, or patch versions.

Friction with SemVer

The impetus for releasing Zarr-Python 3 was to support a new version of the Zarr format. So Zarr-Python 3 was released with a lot of new APIs; some of these APIs have bugs or warts.

For example, we released some functions that should have consistent default values, but due to developer error (this author’s error, in fact) those functions are inconsistent. The fix is simple, in terms of code changes, but it requires breaking changes to our public API. According to SemVer, fixing this bug would require releasing Zarr-Python 4, only a few months after we released 3.

Besides fixing bugs in new APIs, we released Zarr-Python 3 with deprecation notices for many old APIs. We would like to eventually remove these routines from Zarr-Python, but we don’t think it would match user expectations if we released a new major version of the library just to signify removing code. Again, this is contrary to SemVer.

Releasing version 4 of Zarr-Python because we changed the default values of a few functions would likely confuse people. Our users assume that major releases Zarr-Python will contain sweeping changes, not minor refinements of public APIs. This suggests SemVer does not actually fit our project very well, so we decided to update the versioning policy accordingly. If you are interested in the developer discussion about this topic, see Github issue 2889.

Our new versioning policy

We were accustomed to thinking about major releases of Zarr-Python as epochal events that offer lots of new functionality to users (like a brand new version of the underlying Zarr format), but also require substantial changes to existing code. In other words, while major releases likely contain backwards-incompatible changes, backwards-incompatible changes on their own don’t really warrant a major release.

Jacob Tomlinson extended this framing to a full versioning scheme, which he calls Intended Effort Versioning, or “EffVer” for short. The basic idea of EffVer is that you version your project according to the expected effort a user will spend in upgrading to that version.

• Major releases should contain changes have the most impact on users, and thus require the most effort to adopt.
• Minor releases can require some adoption effort from some users.
• Patch releases should require no adoption effort from users.

While SemVer indexes code changes by whether they are backwards compatible or not, EffVer indexes changes on how much effort is required for users to adapt to them. Thus EffVer allows us to ship small-but-breaking changes – like changing default values of some recently-added functions – in a minor release, so long as we think these changes will be easy for users to integrate.

Conclusion

We think switching to EffVer is right for Zarr-Python development. It lets us refine newly-added APIs while reserving major releases for epochal changes, like the Zarr-Python 2 -> 3 transition.

If you have any thoughts or concerns about this decision, we would love to hear from you. The best way to reach us is to open an issue or discussion on our Github page. Thanks for your time!

From the Governance documentation:

The Zarr Steering Council (ZSC) members are core developers who have additional responsibilities to ensure the smooth running of the project. ZSC members are expected to participate in strategic planning, approve changes to the governance model, and make decisions about funding granted to the project itself… The purpose of the ZSC is to ensure smooth progress from the big-picture perspective. Changes that impact the full project require analysis informed by long experience with both the project and the larger ecosystem. When the core developer community (including the ZSC members) fails to reach such a consensus in a reasonable timeframe, the ZSC is the entity that resolves the issue.

On October 14th, Ryan Williams, a founding member of the ZSC, stepped back. In the name of the community, the other members would like to thank Ryan for his support and leadership over the past four years and wish him the best in his new pursuits!

In his place, Norman Rzepka will join the council. Norman has been a key driver of the Zarr v3 specification, the zarrita prototype, a zarr-java implementation, and more recently the rewrite of zarr-python for v3.

Please join us in welcoming him! 👏🏽

~ Josh Moore for the ZSC

Join us for third edition of the Get Your Brain Together Hackathon! This exciting event invites neuroimage data generators, image registration researchers, and neurodata compute infrastructure providers to come together for a hands-on, collaborative experience. Register now and be part of this vibrant community working towards creating reproducible, open-source resources that unlock the mysteries of brain structure and function.

This hackathon will focus on advancing OME-Zarr spatial transformations.

Overview of OME-Zarr

OME-Zarr is a cloud-optimized bioimaging file format that enjoys international community support and widespread adoption in neuroscience. It supports large-scale bioimages with spatial metadata, making it a critical tool for scientific research. The current OME-Zarr standard is enhanced by the coordinate transformations draft, which introduces robust support for spatial transformations. This is particularly important for neuroimaging and other scientific imaging practices, as it facilitates:

• Reproducibility and Consistency: Explicit support for spatial transformations ensures consistent application across various platforms and applications. This feature aligns with the FAIR principles, enabling independent researchers to verify results.
• Integration with Analysis Workflows: By treating spatial transformations as a first-class entity within file formats, OME-Zarr allows seamless integration with diverse image analysis workflows, eliminating the need for additional conversion steps.
• Efficiency and Accuracy: Embedding transformations within the file format minimizes the need for re-sampling, thereby reducing sampling errors and preserving analysis accuracy. This standardization is crucial for handling the massive data volumes generated by modern microscopy techniques.
• Flexibility in Analysis: Native support for spatial transformations provides researchers with the flexibility to apply, modify, or reverse transformations as needed, facilitating longitudinal studies, multi-modal imaging, and comparative analysis.

Hackathon Agenda

The hackathon is structured into three key components:

1. Day 1: Tutorial sessions covering the application needs for coordinate transformations, mathematical principles, and current computational standards and tools available in the open-source ecosystem.
2. Day 2: Small working groups will review and propose enhancements to the current coordinate transformations draft and relevant neuroimaging additions.
3. Day 3: Hands-on activities where participants will implement and apply the proposed improvements to the standards.

Event Details

• Dates: Friday, July 26th - Sunday, July 28th, 2024
• Location: Hybrid event at the University of North Carolina-Chapel Hill, via Google Meet videoconferencing, Image.sc Island Gather.Town virtual space, and Image.sc Zulip Chat.
• Cost: Registration is free!

Register now and add the event to your calendar!

~ Matt McCormick

Zarr’s user, developer, and contributor base is growing every day across several scientific domains, including those responsible for mitigating climate change, solving complex biomedical issues, pushing the boundaries of AI development, and more.

National Aeronautics and Space Administration (NASA) is a prominent user and deeply invested in Zarr among the geospatial community. In this blog post, we’d like to highlight the NASA Prediction Of Worldwide Energy Resources (POWER) project, which has been using Zarr for its data storage needs. The POWER project is based at the NASA Langley Research Center (LaRC), which is located in Hampton, Virginia, USA.

Introduction to POWER and Zarr 🎙

The Prediction Of Worldwide Energy Resources (POWER) project is a cornerstone “Energy and Infrastructure” Earth Action Program project. The Project’s mission is to improve learning, decisions, and outcomes in the renewable energy, sustainable infrastructure, and agroclimatology user communities. For any location in the world, the project provides easily accessible, customized, and trusted NASA solar and meteorological data for past, current, and soon future climates. POWER improves the public and private capability for integrating these NASA Earth Observations (EO) and assimilation model data into their workflows by offering a diverse suite of tools and services to access this data. The project provides access to its Analysis Ready Data (ARD) via POWER’s Application Programming Interface (API), Data Access Viewer (DAV), geospatial services, and cloud-enabled data store. POWER offers no-cost, no-account-needed access to all of its tools, services, and data, which lowers the barrier to entry for many users across the globe. Additionally, through each of its tools and services, POWER’s multi-decadal, low-latency, high-accuracy, community-specific datasets are offered in user-customizable units and a wide variety of formats.

POWER is currently using Zarr as a backend data store for our API services that a lot of our users access to implement in their project’s needs. We have made the backend data stores directly available and freely accessible to the public with no use constraints through Amazon Web Services® (AWS®). We also plan to work with digital twins to integrate POWER data directly in for their data modelling. The POWER Project will continue leveraging the Zarr to store and serve data, which includes dynamic updates at Near Real Time (NRT) by the POWER Project’s data processing code base. Zarr enables the POWER Project to more efficiently support its user communities and impact decision-making for government agencies, non–profit organizations, universities, and private companies around the world.

🗄️ Check POWER’s Zarrs on AWS® Registry of Open Data: https://registry.opendata.aws/nasa-power/

A Brief History of Data Archives 📚

POWER’s meteorological parameters, such as temperature, humidity, precipitation, or wind are derived from the NASA’s Global Modeling and Assimilation Office (GMAO) Modern – Era Retrospective analysis for Research and Applications Version 2 (MERRA-2) assimilation model. MERRA – 2 is a version of NASA’s Goddard Earth Observing System (GEOS) Data Assimilation System. This data is available starting back in 1981.

The energy flux parameters, like solar irradiance and cloud properties, are derived from NASA’s GWEX SRB archive and NASA’s CERES SYN1deg and FLASHFlux projects. This data is available dating back to 1984.

POWER’s ability to provide historical data allows users to access variability, make decisions, and conduct analyses based on past and current information. For more information on POWER’s history, please see our documentation.

What format did POWER use before? 🔙

Previously, POWER used a NetCDF file that was structed support temporal access by chunking along the line of dimension, in conjunction with OPeNDAP software used as middleware to support the POWER API’s temporal requests efficiently. While this system met initial requirements to provide daily data, the team had to explore new formats to meet the growing demand for hourly data. NetCDF was no longer efficient for hourly data needs because NetCDFs is a condensed structure which prompted us to assess the use for Zarr since it is segmented.

Why POWER switched? 🔍

The POWER Project selected the Zarr format as our Analysis Ready Data (ARD) format as we were transitioning from a monolithic server architecture to a microservice-based hybrid cloud hosted architecture environment, with the foresight of fully transitioning to a cloud environment. To support the key component of our services endpoints, the efficient and fast distribution of time-series data, we wanted to remove any middleware software, improve data access, and implement higher levels of data compression.

Benefits after switching 💪🏻

Switching to Zarr enabled complete and direct access to the POWER data archives in an Analysis-Ready Cloud Optimized (ARCO) data store. Zarr also allows for asynchronous writing, JSON metadata, and a folder- based structure which allows us to add to the datastore faster and keep the data neatly sorted. Furthermore, POWER is able to utilize the higher level of data compression which reduces the speed for data acquisition. Lastly, being able to load small parts of the data improves efficiency, relevance and reduces costs.

For POWER’s future datastore, the Zarr’s enhanced compression was leveraged recently in initial testing which resulted in orders of magnitude smaller of total storage volume without losing any data precision. The result of this compression was saving on storage costs and increasing read/write performance of the system.

As a part of NASA’s Space Act Agreement with AWS®, this archive is hosted in S3 as part of the Open Data Registry, which provides the data freely to the public.

Future plans for Zarr 🔮

The POWER Project plans to include both a spatial and time series based chunked data structures to better meet user demand for data and fulfill data orders more quickly, to support more efficient access and analysis. To promote further understanding and enable effective search and discovery, the team will develop enhanced slice-based metadata.

POWER plans to use Zarr spatial data stores for ArcGIS Image Services in future data versions to cater to the needs of the GIS community.

~ Zoe Waring, NASA POWER team

Zarr-Python 2.18

Before we get into the 3.0 release, we’ll first cover a few details about the 2.18 release series. The first thing to know is that we will continue to support 2.18 with bug fixes up until the release of 3.0. Additionally, we expect to use the 2.18 series to communicate changes in the Zarr-Python API, which will come in 3.0. For example, this week’s release included a number of new deprecation warnings for parts of the Zarr-Python API that we expect to remove in 3.0 (e.g. exotic stores, experimental v3 API).

What to expect with Zarr-Python 3.0

In mid-2023, we formed a working group to look at modernizing Zarr-Python and, crucially, adding support for the V3 specification. One of the early outcomes of this effort was a design document detailing the plan for a major refactor to the library. The goals for the refactor effort are to:

• Provide a complete implementation of Zarr V3 through the Zarr-Python API,
• Clear the way for exciting extensions / ZEPs (i.e. sharding, variable chunking, etc.),
• Provide a developer API that can be used to implement and register V3 extensions,
• Improve the performance of Zarr-Python by streamlining the interface between the Store layer and higher level APIs (e.g. Groups and Arrays),
• Clean up the internal and user-facing APIs,
• Improve code quality and robustness (e.g. achieve 100% type hint coverage), and
• Align the Zarr-Python array API with the array API Standard.

In late 2023, we started working on the next version of the library, iterating on core concepts and restructuring the code base. While this effort continues today, here are a few highlights that we are particularly excited about:

• New asynchronous APIs across the library, including at the Store, Group, Array, and Codec levels. The ability for Zarr-Python to leverage asynchronous computation will dramatically improve performance in the library, particularly for workloads that depend on data coming from high-latency stores. We expect most users will interact with these classes through a synchronous interface but the asynchronous alternatives will be available for users that can take advantage of them.
• Complete spec-complaint implementation supporting both V2 and V3. Zarr-Python will support reading and writing in either format. Additionally, the V2 and V3 code paths will benefit from the new asynchronous interfaces as well as other performance improvements.
• New plugin interface for codecs. Previously, codec support was required to run through Numcodecs. Going forward, additional codecs may be registered with Zarr-Python using the zarr.codecs Entry Point. While Numcodecs will continue to supply the Zarr-Python project with most codecs, the plugin interface will support the integration of codecs from other libraries.

Release plan

We are still working hard on the 3.0 development branch. You can follow our progress on our GitHub Project Board. In the coming weeks, we expect to move our development to the main branch of the library and make a series of pre-releases.

Get involved

It’s not too late to get involved with the 3.0 effort. The GitHub Project Board provides an up-to-date summary of outstanding issues. If you maintain a library that depends on Zarr-Python, the 3.0.0-alpha release will be a great time to start testing against the upcoming release. Finally, we continue to hold bi-weekly developer meetings to discuss and coordinate work on Zarr-Python. This is an open meeting so please come if you are interested in getting involved. Check out the Zarr community calendar here.

~Joe Hamman

In this blog, I give a very brief overview of each of the topic areas. More importantly, I link out to the open issues, pull requests, discussions, and meeting opportunities for continued development. You can follow these links to both better understand each effort and also to contribute yourself.

Please keep this work going by adding reviews and comments to online conversations and joining any relevant meetings/working sessions.

Zarr Specification

Zarr Specification refers to the specification for a chunked, compressed, N-dimensional array format primarily designed for storing large numerical arrays efficiently. It is commonly used in scientific computing, geospatial, bioimaging, and data analysis contexts.

The specification defines how data should be organized within a Zarr store, including details on chunking, compression, metadata, and other attributes necessary to efficiently store and retrieve array data. This specification helps ensure interoperability between different software implementations that support the Zarr format.

The latest version of the specification is V2, but Version 3 is in the works.

Chunk Manifest / Virtual Concatenation

In this breakout session, the group engaged in a long technical discussion about a way to define arrays in a Zarr store as concatenations of other arrays in the store. You can read a full ZEP-like description of the discussion here. Shoutout to Tom Nicholas for documenting this so well!

Zarr-Python

Joe Hamman led a group focusing on enabling support for V3 in Zarr-Python. This was part of an ongoing effort working toward Zarr-Python version 3.0 (roadmap). The focus of this group was on closing outstanding issues on the roadmap and testing the development branch in common geospatial applications. Zarr-Python has traditionally been the canonical implementation of Zarr, and it is therefore a current priority since this effort delivers immediate impact to the largest swath of users, including those that use Zarr through downstream libraries (e.g. Xarray, Dask, Anndata, etc.).

Geospatial Multi-Scales/Pyramids

In the Zarr pyramids breakout group, Thomas Maschler and Max Jones discussed the motivations for following the OGC TileMatrixSet 2.0 specification within the GeoZarr specification, which will be shared as a new issue to supersede GeoZarr Issue #30. They also discussed reading those TMS into rio-tiler using Xarray and started a refactor of ndpyramid to support the TMS specification.

Alternate backend for reading remote Zarr stores

Kyle Barron worked on a prototype for an alternate store for Zarr Python using new async Python bindings to Rust’s object-store project. You can see a prototype of object-store-based store implementation at zarr-python#1661.

GeoZarr Specification

Throughout the sprint, the GeoZarr focus group worked on examining the interoperability of GeoZarr and different existing tooling and store support. You can see the table here.

One of the biggest realizations was that ArcGIS has a lot of existing support for Zarr, which is really exciting news! For other tools, there is still work to be done, especially for GeoTIFF-like data being stored in Zarr, which translates to updates needed within the GeoZarr specification. For example, there are functionality issues tied to support or lack thereof for specific compression algorithms. The GeoZarr Steering Working Group is working on providing a list of supported compressions for commonly used tools. There is also work to be done on specifying the organizational structure of GeoZarr and understanding where requirements from CF diverge from the Zarr data model. For this, we are focusing efforts on involving folks with CF expertise to guide these conversations.

If you are interested in helping out, please join the next bi-weekly GeoZarr meeting every other Wednesday at 11 EST. The next will be April 17th and you can find the invite on the Zarr calendar or join directly from this link. Check out the notes from past meetings at the hackmd.

HTTP Extension

A final priority of the Zarr Sprint was to get efforts rolling on how to better visualize Zarr on the web.

Kevin Booth is the lead on this effort. Currently, he has added some sidecar files with links to reference parent, child, and root relationships in the Zarr to be able to use something like traverzarr, the first attempt at traversing a Zarr JSON as if it were a filesystem in developed by Xavier Nogueira during the sprint, to navigate a Zarr in a manner like the Spatio-Temporal Asset Catalog (STAC). A more detailed blog post with updates on this work to come in the next week.

This work continues to be worked on after the sprint. Cloud-Native Geospatial Foundation has started holding bi-weekly meetings to hack on this work. If you would like to be involved in this, email hello@cloudnativegeo.org to be added to the meeting invite, or find the meeting link at the Zarr calendar here.

More efforts to come!

It was great to get a group of people together to spend some dedicated time on Zarr, and the work is nowhere near done. Please help keep the momentum of these efforts going by responding to any GitHub Pull Requests, Issues, or Discussions that you have opinions on and joining any of the established Zarr meetings that are of interest to you. Again, the Zarr calendar can be found here.

~ Michelle Roby

We’ve got an exiciting announcement for you all! 😄

Community Update: Moving to Zulip 💬

We’re excited to announce that the Zarr community has made a move from Gitter to Zulip as our primary chat platform. This transition marks a new chapter for our community and offers several advantages for our members.

Join here → https://ossci.zulipchat.com/

Why Zulip? 🤔

Zulip offers a robust and versatile platform for communication and collaboration. Its threading model allows for organized and focused discussions, making it easier for community members to follow and participate in conversations effectively. Additionally, Zulip provides powerful search capabilities, ensuring that valuable information shared in the past remains accessible to all.

Zulip’s unique message sharing feature allows conversations to be easily shared around the web via unique links. In addition, Zulip’s indexing of all content by search engines ensures that the knowledge base is easily accessible to all users.

Hosting Thanks 🙏🏻

We extend our sincere gratitude to the good humans at the Open Source Science Initiative (OSSCi) for generously hosting the Zulip server. Their commitment to supporting open science and collaborative research is commendable, and we’re thrilled to partner with them on this endeavour.

Shoutout to Jonathan Starr for helping us! 🙌🏻

Building a Hub for Open Science 🧬

The OSSCi Zulip server will serve as a hub for various projects in the scientific Python ecosystem, starting with Zarr. By centralising communication within this platform, we aim to foster greater collaboration, knowledge sharing, and community building among like-minded individuals passionate about open science and research.

Official Chat Platform ™️

With this migration, the OSSCi Zulip server becomes the official chat platform for the Zarr community. We encourage all Zarr users, contributors, and enthusiasts to join us on Zulip to stay updated on the latest developments, seek assistance, and engage with fellow community members.

Your Feedback Matters 🔁

At Zarr, we value the input and ideas of our community members. We’re committed to continuously improving our platform and user experience. Therefore, we welcome any feedback, suggestions, or ideas you may have regarding the Zulip migration or any other aspect of our community. Your input helps us better serve the needs of our users and advance our shared goals.

Please create an issue in zarr-developers/community or join one of our community meetings if you’d like to chat with us!

Join Us on Zulip! 🔗

Ready to join the conversation? Head over to the OSSCi Zulip server and dive into discussions surrounding Zarr and other exciting projects in the scientific Python ecosystem.

With our shift from Gitter to Zulip, it’s worth mentioning that the majority of discussions on Zulip have involved the core developers of Zarr. Now, we’re extending our warm invitation to the wider community to join us on Zulip. Your involvement is crucial as we foster a more inclusive and vibrant community.

We look forward to connecting with you there! ✌🏻

~Sanket Verma

Recently, I and several community members have been speaking at various conferences and events. There has been an exciting development in the Zarr ecosystem, like finalising V3 specification, submitting new ZEPs, initiating new implementations, etc.

While I’m mostly giving beginner talks on Zarr, which answers how, why, and what, the enthusiastic community members have been talking about other exciting stuff!

In this blog post, I highlight a few talks which were delivered in the past two months. Also, we’re maintaining a playlist on YouTube, which has a more extensive collection of talks from various domains and diverse speakers. Check the playlists: Zarr: Introductory Talks and Zarr: Projects, Uses, Research and Workflows.

PyCon DE and PyData Berlin 2023 🇩🇪

I went to Berlin, Germany, in April to speak at PyCon DE and PyData Berlin 2023. My talk was titled “The Beauty of Zarr”, where I emphasised the inner workings using some near illustrations by Trevor Manz. I highlighted how simple, convenient and hackable it is to use Zarr. After going through various explanations, I focused on some critical issues that Zarr eradicates because of its design and workings, i.e. chunking, compression, cloud-enabled etc.

Towards the end, I prepared a Jupyter notebook where I walked through Zarr 101 code to create, read, write and manipulate arrays. I also converted the Zarr pixelated logo from .png to .zarr format, which was a neat closing for my talk.

The slides and notebook can be accessed here.

Please watch the video here: 👇🏻

ESIP Meetings 🌏

Earth Science Information Partners (ESIP) is a community of data and information technology practitioners working together to coordinate earth science interoperability efforts. ESIP has various collaboration areas. ESIP Collaboration areas are made up of administrative committees and small working groups that are called clusters. Some of them are:

• Agriculture & Climate
• Open Science
• Cloud Computing
• Soli Ontology and Informatics
• Data Management Training Clearinghouse
• Council of Data Facilities

And many more.

The ESIP Cloud Computing Cluster organised a three-part series on Zarr titled “Zarr: The Next Generation” In every part, the Zarr Community members talked about several things ranging from V3 to conventions to ZEPs.

The first part took place on March 27th where:

• Ryan Abernathey presented on the Zarr V3 Specification, i.e. ZEP0001
• Martin Durant presented on the variable chunking, i.e. ZEP0003

The video recording of the session can be seen here: 👇🏻

The second part took place on April 24th where:

• Briana Pagán spoke about the current state of GeoZarr specification and working group
• Norman Rzepka spoke about the Sharding specification, i.e. ZEP0002

The video recording of the session can be seen here:

The third part took place on May 22nd where:

• Hailiang Zhang presented the accumulation proposal, i.e. ZEP0005
• Max Jones spoke about Kerchunk and Pangeo-Forge recipes developments

The video recording of the session can be seen here:

These meetings covered a great deal of recent developments in the Zarr ecosystem. The ZEPs mentioned above explained the V3 specification, sharding, and a couple of new exciting features the community is working on. The interesting thing to note here is that the ZEP0003 and ZEP0005 are something the community members wrote to support their use-case in their domain. This shows the openness and flexibility of the Zarr open-source community and how we support everyone. Though these ZEPs are still in the draft state, they’ll be finalised soon for adoption.

I will discuss about V3 specification in a separate blog post, so I’d not go into the details here. But it’s worth noticing GeoZarr specification and what Briana presented. GeoZarr is one of the conventions on top of Zarr specification, which support various use cases of the geospatial community on how they store their data and metadata. The GeoZarr SWG (Steering Working Group) has been working quickly despite the roadblocks (as mentioned by Briana). The progress and specification can be seen here.

Conclusion

These are some of the public engagements done by the Zarr Community members in the past months. If you spoke on Zarr recently or in the past and would like me to highlight your talk, please don’t hesitate to contact me. If you’re working on something interesting which involves Zarr and want to share it with the community, please say ‘Hi’ to me!

I’ll be talking to you all soon.

Until next time, peace! ✌🏻

~Sanket Verma