I’m developing a solution similar to the Copernicus Browser, offering the flexibility to define on-the-fly data processing “recipes” and share the results via WMS (or OGC API - Maps). Everything is based on GDAL VRTs and built-in pixel functions (with plans to support user-defined Python code in the future).
Since baseline 5.12, CDSE has included TLM markers within Sentinel-2 JP2 files. This has brought a dramatic improvement in data access speeds. Unfortunately, this is only the first step toward making CDSE truly cloud-optimized.
I want to generate TLM markers for older baselines (at least those >= 5.00). This would solve the issue for Sentinel-2. However, there is still the challenge of accessing other missions in an optimized manner. I cannot alter the original files, but I can create “proxy files” (like these markers) and include them in the CDSE STAC as additional assets, separate from the regular product data.
My question is: Are you familiar with any workflow for extracting this kind of information from the data? I know that Kerchunk exists for Sentinel-3 and Sentinel-5. I assume I need to go through every .nc file, compute those Kerchunk headers, put them on a private S3 bucket, and link them within the catalog.
I don’t believe there’s a sidecar or proxy file mech for TLM, but it was discussed here with ability to use /vsisparse to insert the XML on the fly:
as for generating them, with GDAL I think you’d have to copy (e.g. CreateCopy(,options=[“TLM=YES”) … ) the file and then extract, then vsisparse them over the original sources (but there might be better ways) -i.e bare example code in the autotest suite gdal/autotest/gdrivers/jp2openjpeg.py at e2b913be67cda104a84aa1f9449cd9479707f4d0 · OSGeo/gdal · GitHub I don’t really understand the relation to kerchunk/virtualizarr/vsikerchunk here, TLM is a similar concept but not the same as chunk byte refs. Happy to discuss/explore though, TLM is a new concept to me.
