2 · What is Zarr?

You've seen why data at this scale needs somewhere better to live. So what exactly is Zarr? Let's start with the shortest honest answer: Zarr is a way to store large, chunked, compressed, N-dimensional arrays so that you can read just the parts you need, from wherever they happen to live. That sentence is doing a lot of work, so let's unpack it one word at a time.

An array is a grid of numbers: the temperature at every point on a map, the brightness of every pixel in a stack of microscope images, the price of every stock at every minute. N-dimensional means the grid can have as many dimensions as your data needs: a time series is one-dimensional, an image is two, a stack of images over time is three or more. For those familiar with Python, this is just a NumPy array so far.

The interesting words are the other two. Chunked means the array is cut into rectangular blocks, and each block is stored on its own. Compressed means each block is squeezed down before it's written. Together these are the machinery behind the promise from Why we need Zarr: you never have to load the whole thing, because the data is already in pieces, and you can fetch only the pieces a given task touches.

Zarr is a protocol, not a file

Here is the idea that the rest of this book leans on, so it's worth slowing down for.

When you see Zarr "data" in the wild, it is tempting to think of it as a file, something you open like a spreadsheet or a PNG. It is more accurate, and far more useful, to think of Zarr as a protocol for laying array data out as a set of key → bytes entries. Each chunk becomes one entry. Some metadata describing the array's shape and type becomes another. The whole array is just a bag of named byte blobs that follow agreed-upon rules.

Why does that distinction matter? Because once the data is "a bag of named blobs," the bag can live anywhere that can store names and bytes: a folder on your laptop, a ZIP file, your computer's memory, or a bucket in the cloud. Nothing about the array changes, only where the blobs are kept. We'll make this concrete in The core idea, but plant the idea now: Zarr describes a layout, and many different backends can hold that layout.

A first picture

Imagine a small 2-D array, say temperatures on a 6×8 grid. Cut it into blocks of 3×4. You now have four chunks. Written out as a Zarr array, that's roughly:

temperature/zarr.json      ← metadata: shape (6, 8), chunks (3, 4), dtype, codecs
temperature/c/0/0          ← top-left chunk, encoded as bytes
temperature/c/0/1          ← top-right chunk
temperature/c/1/0          ← bottom-left chunk
temperature/c/1/1          ← bottom-right chunk

That's it. One small metadata document and four byte blobs, each with a name. You'll meet this exact example again, as an interactive diagram, in The core idea.

Where we're headed

That word "agreed-upon rules" is hiding a whole world — the specification, the proposals that change it, the council that stewards it, and the many libraries that implement it. We'll get there in Parts II and III, after Part I has built the data model those rules describe. First, though, we need to make sure a handful of words mean the same thing to you as they do to the people writing them. That's the next chapter, and everything after it leans on those definitions.