Title: “Firecube: A plugin-based batch ingestion engine for creating operational analysis-ready data cubes”
Speakers: Armagan Karatosun and Anna-Lena Erdmann
When: Wednesday, October 14, 2026 at 12 PM EDT (2026-10-14T16:00:00Z)
Where: Launch Meeting - Zoom
Details:
Join us next Wednesday, October 14 at 12 PM EDT for the next Pangeo showcase presentation!
Abstract:
At EUMETSAT we are working with petabytes of satellite data and climate data records, and our task was simple (!): to turn them into analysis-ready, cloud-optimized Zarr cubes, and to keep doing it as new data arrives. The usual answer, a to_zarr script per dataset, broke down in production: a six-hour ingestion interrupted in its third hour by a virtual machine that died under disk pressure. Appends were serialized on one writer; a crashed job meant starting over, and nobody could tell which slots had been written or by which run.
This is why we built Firecube: a plugin-based ingestion engine that makes those cubes operational, meaning safe to interrupt, resumable, reproducible, and parallel to write, all in native Zarr. A plugin describes the product; Firecube plans the store, coordinates the writers, and keeps a record of what was written. We’ll walk through how it parallelizes writes within one Zarr store, the small control plane that makes runs resumable, and the numbers from a production ingest.
By open-sourcing Firecube, we are looking for users, contributors, collaborators, and feedback so that it serves Zarr cube producers across the scientific community, well beyond EUMETSAT.
Agenda:
- ~15 minutes - Showcase presentation
- 10 - 30 minutes - Discussion
- 15 - 30 minutes - Community check-in