4 · From zero to Zarr

From a NumPy array to stored bytes, chunk by chunk.

This chapter opens Part I, where we build Zarr's data model up one idea at a time. You don't need to know NumPy, HDF5, or anything about file formats. By the end of Part I you'll understand how Zarr stores an array, why that layout is defined by a written specification, and how a library turns those stored bytes back into an array you can use.

We made the case for Zarr back in Why we need Zarr: a firehose of data far too big for memory, and the trick of splitting it into chunks you can fetch and decompress in parallel. This is where that promise becomes concrete.

Part I comes in two more chapters:

• The core idea. The happy path, with pictures and no code.
• Under the hood. A few deeper sections that go off the happy path. Each one is signposted, so you can read on or skip ahead.

Strip away the domain (mosquitoes, galaxies, hurricanes) and the object at the center is always the same: an array, a big grid of numbers. So that's where we begin: with what an array is, then what happens when one grows too big to fit in memory, and from there to how Zarr stores it.