# What's the best file format to chose for raster imagery and masks products
**URL:** https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555
**Category:** Data
**Created:** [October 1, 2024, 7:00am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555 "2024-10-01T07:00:48Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@geynard](https://discourse.pangeo.io/u/geynard)
#### Post date: [October 1, 2024, 7:00am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/1 "2024-10-01T07:00:48Z")
Okay, it’s been some time I wanted to ask this question and get some feedbacks, and this has come up in recent discussions here, so let’s go!
On our imagery production projects (at CNES, French space agency), we often loop on this question: **to which format should we write our products?** This is basically **CoG** vs **Zarr** , with sometimes **NetCDF**. There is probably no good unique answer…
Some advantages and drawbacks I have in mind so far:
- CoG: Well understood by remote sensing community. One file per band, chunked in each file. No parallel writes inside a file, parallel reading. Overviews (nice for visualizations).
- Zarr: More all around. One file per band and chunk (too many files?). Parallel writes and reads. No overviews or by multiplying files?
- Zipped Zarr: Solution to the too many files problem (that can be heavy on infrastructure), solution to how to download one product too, but I feel it is not nice.
- NetCDF: Well understood by many communities. One file per product. Parallel reading using kerchunk?
As said elswhere, (Zipped) Zarr will be the next Sentinel 2 format. I feel there is no strong consensus and good choice on the field of file format for a collection of remote sensing products.
Maybe GeoZarr will change the game here, or Zarr v3 with sub chunks I heard about?
Maybe we got it all wrong and in the future should think more broadly as a collection, a unique Zarr store for every products?
---
### Author:  [@sotosoul](https://discourse.pangeo.io/u/sotosoul)
#### Post date: [October 1, 2024, 6:23pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/2 "2024-10-01T18:23:59Z")
I think there’s a categorical difference between Zarr and all the other formats you mentioned, and that difference relates to the conceptual difference between _file_ and _data_.
In very simple terms, my conceptual description of _files_ is packaged, portable, and complete, self-sufficient entities, but their “internals” need to be extracted in order to be read — and that’s when you get their _data_. Of the examples above, GeoTIFFs, NetCDFs, even zipped Zarrs are indeed conceptual _files_.
Zarr, however, is not. It resembles something close to a consumption-ready data stream. Its internals still use files but this is a functional property of the Zarr system and it’s typically abstracted from the end user.
Since you’re CNES, I assume that one of your use cases is sharing your products with the general public (?). The fact that you mentioned parallel reading a couple of times makes me thinking, are you exploring a data-as-a-service solution?
I get the impression that the scientific community is primarily a file-based culture. People expect _files_ in order to build their _datacube_ and go on with their processing — but seem to be a little reluctant to accept a ready-made datacube! Not that there are that many… perhaps it’s a vicious circle.
I’d say:
- If you want a fail-safe and compatible solution that is not DaaS and can easily get a REST on top, that would be STAC + COGs. You can’t go wrong with those two technologies combined. Also, why rejecting multiband tiffs?
- If you want to set up a DaaS, array-data (i.e., Zarr) is the answer I think, perhaps with a middleware like Arraylake to take care of multichunking and IO.
- It doesn’t seem to me that NetCDF offers something better than those two solutions above, especially if they are exceeding, say, 100MB in size.
---
### Author:  [@geynard](https://discourse.pangeo.io/u/geynard)
#### Post date: [October 1, 2024, 6:47pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/3 "2024-10-01T18:47:41Z")
I think I mainly agree with your introduction. I’ll extend it a bit, Zarr is data, and so can be an entire dataset/collection.
> [@sotosoul](#):
>
> Since you’re CNES, I assume that one of your use cases is sharing your products with the general public (?). The fact that you mentioned parallel reading a couple of times makes me thinking, are you exploring a data-as-a-service solution?
You are right, I thought afterwards that I should have explained the use cases, so here are the main ones for end users:
- Downloading products. It is generaly thought as “one or several products”, like several Sentinel products (files). But I think it should be extended as download a geo-temporal zone of a collection (data). Maybe that’s already two use cases, first one is the most used (but not the better ?).
- Visualize data/products. Well, Google Map/Earth like, without having to transform them. So basically WMS protocol and the likes. There can be other visualization functionnalities, like time series over a point/area of a given variable…
- Analyze data, at scale. With a Cloud or HPC system close to the data (HPC at CNES), Pangeo style workflow on an entire time serie on a big area (see [[WIP] Add satellite image processing benchmark by jrbourbeau · Pull Request #1550 · coiled/benchmarks · GitHub](https://github.com/coiled/benchmarks/pull/1550) typically).
> [@sotosoul](#):
>
> I get the impression that the scientific community is primarily a file-based culture. People expect _files_ in order to build their _datacube_ and go on with their processing — but seem to be a little reluctant to accept a ready-made datacube! Not that there are that many… perhaps it’s a vicious circle.
Yes, I think that it’s more an habit or default. Some colleagues in research really like OpenEO like API to build their datacubes as Zarr and then using Pangeo approach to better analyze them. I think things like stackstac are also really powerfull to abstract files.
> [@sotosoul](#):
>
> You can’t go wrong with those two technologies combined. Also, why rejecting multiband tiffs?
I agree, but not the choice of ESA apparently, which looks like zipped Zarr. And for multiband tiffs, I don’t know, I’ve just never seen it, maybe for download optimization purpose?
> [@sotosoul](#):
>
> It doesn’t seem to me that NetCDF offers something better than those two solutions above
Big advantage is one file for an entire product and metadata. But I wouldn’t go this way for optical raster data or alike neither.
---
### Author:  [@Michael\_Sumner](https://discourse.pangeo.io/u/Michael_Sumner)
#### Post date: [October 1, 2024, 11:12pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/4 "2024-10-01T23:12:46Z")
Excellently put, I have discarded a draft reply I’d started. This is way better
---
### Author:  [@Basile\_Goussard](https://discourse.pangeo.io/u/Basile_Goussard)
#### Post date: [October 2, 2024, 3:13pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/5 "2024-10-02T15:13:12Z")
The easiest solution for us (a startup providing a product based on satellite data) is to rely on STAC + COG. It’s easy to handle and, from my perspective, covers almost all the use cases.
Retrieving data (by selecting the relevant pixels) can be done using `odc.stac` and `rioxarray`.
Visualization can be achieved by relying on `titiler`, as well as by directly adding the URL into QGIS (which works great for quick visualization), and large-scale processing can easily be handled using `coiled` (or HPC).
However, we haven’t succeeded in making it work for one use case:
→ Retrieving the time series of all Sentinel-2 data over more than 60,000 points.
We used `xvec` (a pretty awesome library), but it was still too slow…
> [@Compute time series for 70,000 locations (Speed up the processing)](https://discourse.pangeo.io/t/compute-time-series-for-70-000-locations-speed-up-the-processing/4436/4):
>
> Hi @Basile_Goussard, I have some additional remarks: Do you consider all the time steps or do you filter out cloudy scenes beforehand using the eo:cloud\_cover property? If not applied already, this could save you some time, reducing the number of Items you need to load. Are you running the computation in a serial manner? If you have more than 1 CPU you could easily parallelize the whole computation using libraries like joblib, multiprocessing or dask You could also think about grouping the geo…
I’m not sure if `zarr` would be a better candidate for this use case.
Regarding ML, I don’t know if batching COG (with `xbatch`) would have the same capabilities as `zarr`. I think not, but I’m not sure how many users will try it.
---
### Author:  [@kirill.kzb](https://discourse.pangeo.io/u/kirill.kzb)
#### Post date: [October 2, 2024, 11:31pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/6 "2024-10-02T23:31:01Z")
@geynard
> CoG: Well understood by remote sensing community. One file per band, chunked in each file. No parallel writes inside a file, parallel reading. Overviews (nice for visualizations).
`odc-geo` includes a parallel write method for generating COGs from Dask backed arrays.
[https://odc-geo.readthedocs.io/en/latest/\_api/odc.geo.cog.save\_cog\_with\_dask.html#odc.geo.cog.save\_cog\_with\_dask](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.cog.save_cog_with_dask.html#odc.geo.cog.save_cog_with_dask)
It’s capable of writing large outputs directly to S3, and does so in a single pass over data, including overview generation while following all the internal file structure constraints of the COG. It is not limited to a single machine either, both compression and writing out to disk or S3 are done concurrently taking advantage of available compute in the cluster.
---
### Author:  [@strobpr](https://discourse.pangeo.io/u/strobpr)
#### Post date: [October 4, 2024, 1:58pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/7 "2024-10-04T13:58:56Z")
Very interesting discussion! I guess you are aware that this question is currently kicked around at ESA for their what they call the EO processing Framework ([EOPF](https://eopf.copernicus.eu/eopf/)) for Copernicus. It seems that Zarr is the format of choice but some implementation questions (e.g. pyramids) are still open, not sure to what extent zipping is applied, more info is here: [Product Structure and Format Definition — EOPF - Core Python Modules](https://cpm.pages.eopf.copernicus.eu/eopf-cpm/main/PSFDjan2024.html#product-structure-and-format-definition). It would be extremely useful to receive as much feed-back as possible also from people outside the Copernicus bubble on these imminent decisions which will shape the way Petabytes of Sentinel Data will be thrown at us for the rest of this decade at least.
There will be a workshop end of November at ESA-ESRIN which is probably one of the last occasions to provide feed-back which has an impact on this important scheme.
---
### Author:  [@geynard](https://discourse.pangeo.io/u/geynard)
#### Post date: [October 4, 2024, 8:29pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/8 "2024-10-04T20:29:29Z")
> [@kirill.kzb](#):
>
> `odc-geo` includes a parallel write method for generating COGs from Dask backed arrays.
Do you mean that you are able to write one COG file in parallel on S3 or on a standard file system with Dask and Xarray? How does that work? This would change the conclusion of the EOPF definition trade-offs shared by @strobpr.
> [@strobpr](#):
>
> I guess you are aware that this question is currently kicked around at ESA for their what they call the EO processing Framework ([EOPF](https://eopf.copernicus.eu/eopf/)) for Copernicus
This has come up several times, so yes. At CNES we had a bit of contact several months/years ago, but we lost track of the advances here. Thanks or the link to the current version! I was really happy at seing an earlier version of it, as I share most of the objectives of this new format. But the thing is I currently don’t see any clear answer, at least for Sentinel 2 SAFE like optical products.
> The zarr format stores a block in a file and can be read and written with block-wise parallelisation. It is currently considered the default format for EOPF product items because it supports both reading and writing from cloud storage.
> The zipped zarr is a container format for the multi-file zarr that packages the zarr files into a single non-compressed zip file. This can be transferred, and it can also be stream-accessed in cloud storage (or local storage) for reading without unzipping. It is expected to be a preferred transfer format.
> [@strobpr](#):
>
> There will be a workshop end of November at ESA-ESRIN which is probably one of the last occasions to provide feed-back which has an impact on this important scheme.
Do you have more information about this event?
---
### Author:  [@rabernat](https://discourse.pangeo.io/u/rabernat)
#### Post date: [October 4, 2024, 8:54pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/9 "2024-10-04T20:54:12Z")
> [@strobpr](#):
>
> It would be extremely useful to receive as much feed-back as possible also from people outside the Copernicus bubble on these imminent decisions which will shape the way Petabytes of Sentinel Data will be thrown at us for the rest of this decade at least.
I’ve heard about this project and rumors that it was going to use Zarr. What is the best way for our community (which contains the core developers of this technology) to provide such feedback?
---
### Author:  [@kirill.kzb](https://discourse.pangeo.io/u/kirill.kzb)
#### Post date: [October 7, 2024, 5:33am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/10 "2024-10-07T05:33:12Z")
> Do you mean that you are able to write one COG file in parallel on S3 or on a standard file system with Dask and Xarray? How does that work?
Correct, `odc-geo` compresses every TIFF block separately. It also constructs Dask graphs for overview images and compresses those too, although one can also supply overviews as independent inputs. Compressed blocks are then concatenated in the order required by the COG format and written out to a final location using multi-part upload functionality of S3. Multi-part upload on AWS supports splitting any object into up to 10k ordered blocks, provided each block is at least 5MiB in size. There is some implementation complexity coming from the interaction of laziness of Dask with minimal block size requirement, but it works without forcing evaluation order or forcing `persist` calls. It is possible to create COGs that are larger than total available cluster memory, as we are flushing compressed bytes to S3 as we go.
---
### Author:  [@strobpr](https://discourse.pangeo.io/u/strobpr)
#### Post date: [November 3, 2024, 10:40am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/11 "2024-11-03T10:40:51Z")
## Apologies for the late reply! I copy below a mail I received from ESA (Jolyon.Martin@esa.int). Some info should also be available here: [STAC Shared Resources - STAC Shared Resources - ESA EO Framework](https://eoframework.esa.int/x/yYEWAQ) (link didn’t work when just tested - hope this is temporary). Contact Jolyon or Angela.Lombardi@ext.esa.int for latest info.
Dear All
At the workshop on STAC we held before the summer [https://eoframework.esa.int/display/SSR/STAC+Workshop±+June+6%2C+2024](https://urldefense.com/v3/ __https://eoframework.esa.int/display/SSR/STAC*Workshop*-*June*6*2C*2024__ ;KysrKyUr!!DOxrgLBm!GzkJMGxGZ3i-CNgixzR6xnJ-3YrxuFLogKPYIz1FpdKEHYtcXi3D79n0kiZSiUzqdKZvD1z1RjtPSF-0hHJI_s_IaOId5A$) we suggested a follow on to be arranged in the autumn. **We would therefore like to propose a “save the date” for Tuesday 26** **November** for this event, we did a preliminary check that there are no major Copernicus happenings at this time so I hope this works well.
From an ESA perspective we would be pleased to present the progress on STAC for the Sentinels data, and the extended use of the STAC within the various elements of the Copernicus Ground Segment. We would also take the opportunity to share a more in-depth look at the work being performed by ESA in preparation of a new common product format to be deployed across the Sentinels, and Copernicus Expansion missions in the future, using ZARR.
As a very rough planning for this second workshop I would propose the morning dedicated more to STAC and the afternoon more on ZARR. For ESA’s part we will send out an information package to allow you all to take a look at the work performed to date and bring any questions, suggestions or comments. Potentially we would also look at hosting a STAC sprint next year, and so some discussion on that at the workshop would be welcome.
The information shared at the last workshop was very welcome, and perhaps this time there would be specific themes that the attendees would like to focus. Please let me know if there is any specific topic you would like to present or discuss so that we can compile a more detailed agenda, we would welcome any input/presentation on the STAC and ZARR ecosystem.
**I am using the email list for those informed of the last workshop, please feel free to forward this notice to whomever you feel could be interested**. Due to the building work ongoing in ESRIN we are a bit limited in hosting people on-site, so this time the meeting would be teams only. **Please let Angela know if you would like to be included or dropped from the email list.**
Kind regards
Jolyon Martin
---
### Author:  [@RichardScottOZ](https://discourse.pangeo.io/u/RichardScottOZ)
#### Post date: [November 6, 2024, 10:05pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/12 "2024-11-06T22:05:59Z")
Zarr zip driver is slower too when I have tried it.
I am considering recommendations on this for proprietary format conversion as well - geophysics grids as such, but there will be other types.
Some of the similar things I have been writing on confluence.
Geoscientists understand geotiffs - at least at a basic level, not a data level - and can use directly.
---
### Author:  [@nunomir](https://discourse.pangeo.io/u/nunomir)
#### Post date: [November 21, 2024, 6:37am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/13 "2024-11-21T06:37:11Z")
It seems that the zarr + stac workshop will be postponed to next year. Likely in February.
---
### Author:  [@csaybar](https://discourse.pangeo.io/u/csaybar)
#### Post date: [January 5, 2025, 12:01pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/14 "2025-01-05T12:01:12Z")
Unfortunately, I’m unable to add links directly as I’m a new user, I don’t have the necessary privileges. However, I’ve outlined below the reasons why I believe Zarr+ZIP is not a good idea: 👇
> **[GitHub - csaybar/ESA-zar-zip-decision: What I think about tentative zarr+zip as the next...](https://github.com/csaybar/ESA-zar-zip-decision)**
>
> What I think about tentative zarr+zip as the next Sentinel 2 format
I hope this helps reconsider some of the decisions regarding the use of the ZIP format. It explains why the ZIP format will always be slow.
---
### Author:  [@Michael\_Sumner](https://discourse.pangeo.io/u/Michael_Sumner)
#### Post date: [February 27, 2025, 4:06am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/15 "2025-02-27T04:06:21Z")
excellent work, and sorry to have missed this … 100% agree this is a disaster and I hope others are speaking up against it
---
### Author:  [@emmanuelmathot](https://discourse.pangeo.io/u/emmanuelmathot)
#### Post date: [March 5, 2025, 5:33pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/16 "2025-03-05T17:33:13Z")
I just spent the last two days at a workshop organized by ESA, where the roadmap for Zarr adoption within the EOPF framework was presented. Below, I will summarize the key takeaways relevant to this discussion until official information is officially released in April:
[EOPF](https://eopf.copernicus.eu/) (Earth Observation Processing Framework) is the abstraction framework for the new Copernicus Space Component (CSC) Data Processor Re-engineering. It serves as a data model that abstracts product items such as measurements, quality, annotations, and attributes. In theory, this framework would allow for encoding products in multiple containers, including Zarr, GeoTIFF, NetCDF, and SAFE.
The selected baseline data encoding for all missions is Zarr, chosen for various justified reasons such as scalability and cost efficiency. The definition of the specifications initiative began in 2024 and will continue until 2026, when ESA is expected to start operationally producing the new Zarr from the ground segment.
By that time, iterations will need to be made regarding the Zarr specification and partitioning. A key point confirmed by all ESA technical officers is that **there will be no Zipped Zarr in the final implementation**. The Zipped Zarr format is only a temporary workaround for managing the enormous number of files generated by the combination of the EOPF data model, partitioning, and Zarr structure for a single data product. This is where contributions like sharding will be welcome.
The Sample Service, developed and operated by EODC, will serve as the central point of reference (official opening by April) for all community activities around EOPF Zarr. More information on the Sample Service can be found here: [https://zarr.eopf.copernicus.eu/](https://zarr.eopf.copernicus.eu/)
Additionally, there will be a STAC + Zarr workshop on April 17th. More information to come.
ESA also aims to implement a Discrete Global Grid System (DGGS) for data mapping to reduce current data duplication, which can be as high as 30% due to Sentinel-2’s current UTM gridding.
It is important to note that this initiative is still underway, and ESA has several key matters to address in collaboration with the community before they can begin producing Sentinel data in Zarr on a permanent basis. The key takeaway is that ESA’s current focus is on promoting community engagement and encouraging user adoption, which is a welcomed effort.
---
### Author:  [@c.defranchis](https://discourse.pangeo.io/u/c.defranchis)
#### Post date: [March 9, 2025, 6:41pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/17 "2025-03-09T18:41:51Z")
Hi Emmanuel,
Thank you for sharing these takeaways! Do you know if the roadmap includes a plan for reprocessing the entire Sentinel-1/2/3 archives to the new format?
---
### Author:  [@Christoph\_Reimer](https://discourse.pangeo.io/u/Christoph_Reimer)
#### Post date: [March 12, 2025, 9:30pm UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/18 "2025-03-12T21:30:51Z")
I’m happy to announce that the first webinar related to the EOPF Zarr data format will take place on the 27 March 2025 at 11:00 CET. Register now via the given [Webinar Registration - Zoom](https://us02web.zoom.us/webinar/register/WN_MzGnEYQKSc6zGYOchzCznQ#/registration) link. Join the webinar and contribute to the specification of this new format. Looking forward to a fruitful discussion.
---
### Author:  [@tinaok](https://discourse.pangeo.io/u/tinaok)
#### Post date: [March 14, 2025, 5:17am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/19 "2025-03-14T05:17:45Z")
Hi c.defrancis,
As I understand, eventually yes, all data will be converted.
---
### Author:  [@tinaok](https://discourse.pangeo.io/u/tinaok)
#### Post date: [April 11, 2025, 8:43am UTC](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555/20 "2025-04-11T08:43:24Z")
Hi everyone,
(may be we should start a new thread specific to updates on EOPF??, but I’ll put this info here;)
The STAC & Zarr Workshop will be held next Wednesday 16.04.25 10:00-16:00 (CEST) It is a online workshop, registration link is;
[https://forms.oXice.com/e/JRpNqA384D](https://forms.oxice.com/e/JRpNqA384D)
This is what I got as draft agenda ;
Agenda – STAC & Zarr Workshop – draft
Morning 10:00 to 12:00
10:00 – 10:10 - Introduction - ESA
10:10 – 10:25 – STAC in the Copernicus Data Space Ecosystem - CF
10:25 – 10:40 - openEO interface using PG STAC - Sinergise
10:40 – 10:55 - Context of Zarr in EOPF re-engineering - ESA
10:55 – 11:05 - STAC in ESA EOPF Zarr - CS
11:05 – 11:20 - EOPF Sample Service - EODC
11:20 – 11:30 - STAC Asset definition for Zarr - DLR
11:30 – 11:40 - STAC for EOPF orchestration – Airbus
11:40 – 12:00 – Open Discussion
Afternoon 14:00 to 16:00
14:00 – 14:15 - ECMWF Data Stores approach to STAC/Zarr - ECMWF
14:00 – 14:30 - Zarr for Sentinel-3 Marine Data - EUMETSAT
14:30 – 14:45 - STAC in ESA Common Services (TBC) – SPACEBAR
14:45 – 15:00 - Transforming research and operational pipelines: STAC at BEYOND - Νational Οbservatory of Αthens
15:00 – 15:15 – STAC & Zarr for Climate Data Cache on DESP – B-Open
15:30 – 15:45 - GeoZarr – Dev. Seed
15:45 – 16:00 – Open Discussion and Conclusion
[Next page](https://discourse.pangeo.io/t/whats-the-best-file-format-to-chose-for-raster-imagery-and-masks-products/4555.md?page=2)