Okay, it’s been some time I wanted to ask this question and get some feedbacks, and this has come up in recent discussions here, so let’s go!
On our imagery production projects (at CNES, French space agency), we often loop on this question: to which format should we write our products? This is basically CoG vs Zarr, with sometimes NetCDF. There is probably no good unique answer…
Some advantages and drawbacks I have in mind so far:
CoG: Well understood by remote sensing community. One file per band, chunked in each file. No parallel writes inside a file, parallel reading. Overviews (nice for visualizations).
Zarr: More all around. One file per band and chunk (too many files?). Parallel writes and reads. No overviews or by multiplying files?
Zipped Zarr: Solution to the too many files problem (that can be heavy on infrastructure), solution to how to download one product too, but I feel it is not nice.
NetCDF: Well understood by many communities. One file per product. Parallel reading using kerchunk?
As said elswhere, (Zipped) Zarr will be the next Sentinel 2 format. I feel there is no strong consensus and good choice on the field of file format for a collection of remote sensing products.
Maybe GeoZarr will change the game here, or Zarr v3 with sub chunks I heard about?
Maybe we got it all wrong and in the future should think more broadly as a collection, a unique Zarr store for every products?
I think there’s a categorical difference between Zarr and all the other formats you mentioned, and that difference relates to the conceptual difference between file and data.
In very simple terms, my conceptual description of files is packaged, portable, and complete, self-sufficient entities, but their “internals” need to be extracted in order to be read — and that’s when you get their data. Of the examples above, GeoTIFFs, NetCDFs, even zipped Zarrs are indeed conceptual files.
Zarr, however, is not. It resembles something close to a consumption-ready data stream. Its internals still use files but this is a functional property of the Zarr system and it’s typically abstracted from the end user.
Since you’re CNES, I assume that one of your use cases is sharing your products with the general public (?). The fact that you mentioned parallel reading a couple of times makes me thinking, are you exploring a data-as-a-service solution?
I get the impression that the scientific community is primarily a file-based culture. People expect files in order to build their datacube and go on with their processing — but seem to be a little reluctant to accept a ready-made datacube! Not that there are that many… perhaps it’s a vicious circle.
I’d say:
If you want a fail-safe and compatible solution that is not DaaS and can easily get a REST on top, that would be STAC + COGs. You can’t go wrong with those two technologies combined. Also, why rejecting multiband tiffs?
If you want to set up a DaaS, array-data (i.e., Zarr) is the answer I think, perhaps with a middleware like Arraylake to take care of multichunking and IO.
It doesn’t seem to me that NetCDF offers something better than those two solutions above, especially if they are exceeding, say, 100MB in size.
I think I mainly agree with your introduction. I’ll extend it a bit, Zarr is data, and so can be an entire dataset/collection.
You are right, I thought afterwards that I should have explained the use cases, so here are the main ones for end users:
Downloading products. It is generaly thought as “one or several products”, like several Sentinel products (files). But I think it should be extended as download a geo-temporal zone of a collection (data). Maybe that’s already two use cases, first one is the most used (but not the better ?).
Visualize data/products. Well, Google Map/Earth like, without having to transform them. So basically WMS protocol and the likes. There can be other visualization functionnalities, like time series over a point/area of a given variable…
Yes, I think that it’s more an habit or default. Some colleagues in research really like OpenEO like API to build their datacubes as Zarr and then using Pangeo approach to better analyze them. I think things like stackstac are also really powerfull to abstract files.
I agree, but not the choice of ESA apparently, which looks like zipped Zarr. And for multiband tiffs, I don’t know, I’ve just never seen it, maybe for download optimization purpose?
Big advantage is one file for an entire product and metadata. But I wouldn’t go this way for optical raster data or alike neither.
The easiest solution for us (a startup providing a product based on satellite data) is to rely on STAC + COG. It’s easy to handle and, from my perspective, covers almost all the use cases.
Retrieving data (by selecting the relevant pixels) can be done using odc.stac and rioxarray.
Visualization can be achieved by relying on titiler, as well as by directly adding the URL into QGIS (which works great for quick visualization), and large-scale processing can easily be handled using coiled (or HPC).
However, we haven’t succeeded in making it work for one use case:
→ Retrieving the time series of all Sentinel-2 data over more than 60,000 points.
We used xvec (a pretty awesome library), but it was still too slow…
I’m not sure if zarr would be a better candidate for this use case.
Regarding ML, I don’t know if batching COG (with xbatch) would have the same capabilities as zarr. I think not, but I’m not sure how many users will try it.
CoG: Well understood by remote sensing community. One file per band, chunked in each file. No parallel writes inside a file, parallel reading. Overviews (nice for visualizations).
odc-geo includes a parallel write method for generating COGs from Dask backed arrays.
It’s capable of writing large outputs directly to S3, and does so in a single pass over data, including overview generation while following all the internal file structure constraints of the COG. It is not limited to a single machine either, both compression and writing out to disk or S3 are done concurrently taking advantage of available compute in the cluster.
Very interesting discussion! I guess you are aware that this question is currently kicked around at ESA for their what they call the EO processing Framework (EOPF) for Copernicus. It seems that Zarr is the format of choice but some implementation questions (e.g. pyramids) are still open, not sure to what extent zipping is applied, more info is here: Product Structure and Format Definition — EOPF - Core Python Modules. It would be extremely useful to receive as much feed-back as possible also from people outside the Copernicus bubble on these imminent decisions which will shape the way Petabytes of Sentinel Data will be thrown at us for the rest of this decade at least.
There will be a workshop end of November at ESA-ESRIN which is probably one of the last occasions to provide feed-back which has an impact on this important scheme.
Do you mean that you are able to write one COG file in parallel on S3 or on a standard file system with Dask and Xarray? How does that work? This would change the conclusion of the EOPF definition trade-offs shared by @strobpr.
This has come up several times, so yes. At CNES we had a bit of contact several months/years ago, but we lost track of the advances here. Thanks or the link to the current version! I was really happy at seing an earlier version of it, as I share most of the objectives of this new format. But the thing is I currently don’t see any clear answer, at least for Sentinel 2 SAFE like optical products.
The zarr format stores a block in a file and can be read and written with block-wise parallelisation. It is currently considered the default format for EOPF product items because it supports both reading and writing from cloud storage.
The zipped zarr is a container format for the multi-file zarr that packages the zarr files into a single non-compressed zip file. This can be transferred, and it can also be stream-accessed in cloud storage (or local storage) for reading without unzipping. It is expected to be a preferred transfer format.
I’ve heard about this project and rumors that it was going to use Zarr. What is the best way for our community (which contains the core developers of this technology) to provide such feedback?
Do you mean that you are able to write one COG file in parallel on S3 or on a standard file system with Dask and Xarray? How does that work?
Correct, odc-geo compresses every TIFF block separately. It also constructs Dask graphs for overview images and compresses those too, although one can also supply overviews as independent inputs. Compressed blocks are then concatenated in the order required by the COG format and written out to a final location using multi-part upload functionality of S3. Multi-part upload on AWS supports splitting any object into up to 10k ordered blocks, provided each block is at least 5MiB in size. There is some implementation complexity coming from the interaction of laziness of Dask with minimal block size requirement, but it works without forcing evaluation order or forcing persist calls. It is possible to create COGs that are larger than total available cluster memory, as we are flushing compressed bytes to S3 as we go.
Apologies for the late reply! I copy below a mail I received from ESA (Jolyon.Martin@esa.int). Some info should also be available here: https://eoframework.esa.int/x/yYEWAQ (link didn’t work when just tested - hope this is temporary). Contact Jolyon or Angela.Lombardi@ext.esa.int for latest info.
Dear All
At the workshop on STAC we held before the summer https://eoframework.esa.int/display/SSR/STAC+Workshop±+June+6%2C+2024 we suggested a follow on to be arranged in the autumn. We would therefore like to propose a “save the date” for Tuesday 26November for this event, we did a preliminary check that there are no major Copernicus happenings at this time so I hope this works well.
From an ESA perspective we would be pleased to present the progress on STAC for the Sentinels data, and the extended use of the STAC within the various elements of the Copernicus Ground Segment. We would also take the opportunity to share a more in-depth look at the work being performed by ESA in preparation of a new common product format to be deployed across the Sentinels, and Copernicus Expansion missions in the future, using ZARR.
As a very rough planning for this second workshop I would propose the morning dedicated more to STAC and the afternoon more on ZARR. For ESA’s part we will send out an information package to allow you all to take a look at the work performed to date and bring any questions, suggestions or comments. Potentially we would also look at hosting a STAC sprint next year, and so some discussion on that at the workshop would be welcome.
The information shared at the last workshop was very welcome, and perhaps this time there would be specific themes that the attendees would like to focus. Please let me know if there is any specific topic you would like to present or discuss so that we can compile a more detailed agenda, we would welcome any input/presentation on the STAC and ZARR ecosystem.
I am using the email list for those informed of the last workshop, please feel free to forward this notice to whomever you feel could be interested. Due to the building work ongoing in ESRIN we are a bit limited in hosting people on-site, so this time the meeting would be teams only. Please let Angela know if you would like to be included or dropped from the email list.