# Recommendation for hosting cloud-optimized data

**URL:** <https://discourse.pangeo.io/t/recommendation-for-hosting-cloud-optimized-data/2063>\
**Category:** Data\
**Created:** [January 12, 2022, 8:17pm UTC](https://discourse.pangeo.io/t/recommendation-for-hosting-cloud-optimized-data/2063 "2022-01-12T20:17:52Z")\
**Posts on this page:** 1\
**Showing post:** 11

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [January 21, 2022, 1:42pm UTC](https://discourse.pangeo.io/t/recommendation-for-hosting-cloud-optimized-data/2063/11 "2022-01-21T13:42:57Z")

</div>

Apologies for not updating with the new Zarr files to match the NetCDF chunking. I ran into [Debugging memory issues in IMERG · Issue #227 · pangeo-forge/pangeo-forge-recipes · GitHub](https://github.com/pangeo-forge/pangeo-forge-recipes/issues/227) and haven’t had a chance to dig into it.

I’m glad to hear that others have already done similar benchmarks and have reached the conclusion that performance is roughly similar (which make sense, but it’s good to see confirmed).

> [Rich] Do we know how many concurrent requests it would take before we would see less than linear scaling?

[https://docs.microsoft.com/en-us/azure/storage/blobs/scalability-targets](https://docs.microsoft.com/en-us/azure/storage/blobs/scalability-targets) has some numbers. We haven’t observed slowdowns, but we / our users have hit the limits off the blob storage service, and so have had to add retry logic to their applications.

> [Ryan] **the primary data store on the cloud should be the original netCDF files + a kerchunk index**.

I like this recommendation, primarily because it means I’m not in the awkward position of claiming we host dataset X, when it’s actually a somewhat different (but better!) cloud-optimzied version.

> [Ryan] How should a sequence of netCDF files + kernchunk index be represented in STAC? Does each individual netCDF file need to go into the STAC catalog? Or does the kerchunk index effectively serve as a catalog for the individual files?

Anything is possible, but I one factor is whether you’re exposing both the  
netCDF files _and_ the Kerchunk index through the STAC API. If you’re making  
both available (as I think you should), I’d recommend:

1. Use STAC Items + assets to catalog the netCDF files.
2. Use collection-level assets (or a separate STAC collection) for Kerchunk  
index files

Otherwise, you risk searches returning “duplicate” matches, one for the regular  
netCDF files and one for the index file.

> Do we need a formal kerchunk STAC extension?

I’m not sure yet. I’ve found the STAC extensions helpful to enable tools like  
`intake-stac` be able to programatically, and safely, go from STAC → Dataset.

Currently, loading these kerchunk-style datasets looks like:

```python
import requests

asset = collection.assets["ACCESS-CM2.historical"]

references = requests.get(asset.href).json()

reference_filesystem = fsspec.filesystem("reference", fo=references)
reference_filesystem = planetary_computer.sign(reference_filesystem)

ds = xr.open_dataset(
    reference_filesystem.get_mapper("/"),
    **asset.extra_fields["xarray:open_dataset_kwargs"]
)

```

The new thing here is that intermediate `requests.get(...)` to get the  
references (I think _we_ can’t quite just pass the URL, since the Azure Blob  
Storage URLs need to be signed, but other groups might be able to skip that).  
So the only additional thing needed might be a flag in the xarray metadata  
indicating that it’s a kerchunk file / reference filesystem, so that tools  
like intake-stac know what to do.

> [Ryan] Can we use checksumming to verify the file integrity?

There’s a few places we might want to do that:

1. From the original data provider (e.g. a THREDDS server) to Blob Storage. If  
the data provider provides checksums, we should validate against them.
2. The kerchunk index: to verify that the data read via kerchunk / reference  
filesystem is identical to the data you would have gotten via the NetCDF file

On this second item, what exactly would we be checksumming / comparing? That the  
raw bytes of the NetCDF variable is identical to what we’d we read via kerchunk?  
Or is the range request made by Kerchunk just a subset of the variable’s data  
(skipping a header or footer, for example).

As for storing those checksum values, I think either the STAC metadata, or  
perhaps within the Kerchunk index file itself would be fine.

---

_[View the full topic](https://discourse.pangeo.io/t/recommendation-for-hosting-cloud-optimized-data/2063)._