1 · Why we need Zarr

Before we say what Zarr is, it's worth a page on why it needed to exist at all. Every idea in this book is easier to hold onto once you know the problem it was built to solve.

In short

Modern instruments and simulations produce arrays of numbers far too big to fit in memory, and that data needs to be stored durably and shared widely. Zarr stores these giant arrays so a reader can cheaply fetch just the piece they want, and so the work of reading them can run in parallel across many CPU cores.

A firehose of numbers

Across science and industry, our instruments and simulations have become extraordinary firehoses of numbers. A satellite streams images of the Earth. A microscope captures gigapixel scans. A gene sequencer reads thousands of genomes. A climate model writes out temperature and wind for every point on the globe, hour after hour. In each case the result has the same shape: a vast grid of numbers, far more than fits in any single computer's memory, and often arriving as a continuous stream.

That data is worth little sitting on one machine. It has to be stored somewhere durable and shareable, so that many people (often scattered across the world) can read and analyze it. Increasingly that somewhere is cloud object storage (such as Amazon S3, Google Cloud Storage, or Azure Blob Storage): cheap, effectively unlimited, and reachable from anywhere. But sheer size makes this hard. Nobody wants to download terabytes just to inspect one corner. What's needed is a way to store these giant grids so a reader can efficiently and cheaply fetch just the piece they want.

Where Zarr came from

Zarr was built to solve exactly this, though it didn't begin with the cloud. It grew out of genomics. Around 2015, Alistair Miles needed to analyze arrays of genetic variation across thousands of malaria-carrying mosquitoes (the Anopheles gambiae 1000 Genomes Project), arrays far too big to fit in memory. The real frustration was speed, and to see why, it helps to understand two things the array formats of the day were already doing: chunking and compression.

First, chunking. To store an array bigger than memory, formats like HDF5 and netCDF already split it into blocks (called chunks) and compress each one. That's what lets you read part of an array without loading all of it: you only fetch and decompress the chunks that cover the part you want. None of this is Zarr's invention. Chunking and compression were well-established ideas, and Zarr deliberately reuses them. The catch with the existing tools was speed: decompression takes CPU work, and for a big analysis that scans millions of values, that work adds up fast.

Here's where speed came in. Reading a chunk means decompressing it, so reading many chunks is a pile of independent decompression jobs — exactly the kind of work you'd want to spread across all your CPU cores at once. But the tools of the day wouldn't allow it. In Python, the global interpreter lock (GIL) limits how much work threads can do at the same time, so reading through HDF5 couldn't keep all the cores busy. And the other chunked format available could split an array along only its first dimension, while scientific arrays usually have several dimensions. Those analyses kept needing pieces that cut across those dimensions, and chunking along just one of them made that painfully slow. One core did all the work while the rest sat idle.

So he built Zarr. It didn't introduce new storage concepts so much as recombine familiar ones (chunks, compression, metadata) in a way that frees the CPU cores to work in parallel: cut an array into chunks across all its dimensions at once, not just one, and decompress them concurrently. Now a read becomes many chunk-decompressions running at the same time, across every core on the machine (and, with tools like Dask, across many machines), so an analysis that crunches the whole array finishes in a fraction of the time. He tells the story in his early Zarr blog posts.

Then came the cloud

Storing data in the cloud came later, and turned out to be a superpower. Because each chunk is simply one key/value entry (as we'll see), Zarr maps naturally onto object storage like S3, which made it a backbone of cloud-native science. Today Zarr is used far beyond genomics: in Earth and climate science (satellite imagery and weather and climate model output, via the Pangeo community), bio-imaging (huge microscopy volumes, via OME-Zarr), astronomy, and machine learning — anywhere people wrestle with large, multi-dimensional grids of numbers.

Strip away the domain (mosquitoes, galaxies, hurricanes) and the object at the center is always the same: an array, a big grid of numbers. So that's where we begin — with what an array is, and what Zarr actually does with it.