Icechunk: A new cloud-native transactional storage engine for Zarr

Hi everybody!

Our team at Earthmover is excited to announce a brand new open source project called Icechunk. Here’s our launch blog post:

Icechunk is an open-source library and specification for the storage of multidimensional array (a.k.a. tensor) data in cloud object storage. Icechunk works together with Zarr, augmenting the Zarr core data model with features that enhance performance, collaboration, and safety in a multi-user cloud-computing context. With the release of Icechunk, powerful capabilities such as isolated transactions and time travel, which were previously only available to Earthmover customers via our Arraylake platform, are now free and open source.

We’re super excited to get feedback on Icechunk from the Pangeo community.

You can learn more about Icechunk on the docs page:

and on GitHub

If you’re excited about this project, please help us out by giving Icechunk a :star: on GitHub! :pray:

21 Likes

Those performance numbers are exciting :slight_smile:

5 Likes