What are the best practices to store data in zarr and how scalable is zarr?

[TomNicholas]

@Kaboom_Official Zarr / Icechunk can scale pretty large.

This GOES-16 archive has 7 billion chunks

GOES-16 as a single virtual Zarr store

I stress-tested VirtualiZarr and Icechunk by ingesting all of the GOES-16 Cloud and Moisture Imagery Product archive into a single store. As far as I know this is the biggest virtual Zarr store ever (by number of chunks - 7.1 billion).

and there are several copies of ERA5 in Zarr, each at Petabyte-scale

Low Latency Icechunk ERA5 Now Available on the Earthmover Data Marketplace |...