Parallel creation of STAC catalogs

Data
2

Hi @keewis . Hopefully I can provide a bit of background for this based on our previous experiences. I’m often working closely with data providers who are generating large numbers of STAC Items representing all of their data holdings. In the initial days of STAC the idea of hierarchy of linked, static parent/child files in object storage that would be easily crawlable for indexing seemed fairly reasonable. But as the scale the collections we were managing grew both in size and frequency of data additions and removals, it became clear that the static, hierarchy model would be difficult to manage.

  1. Making atomic updates to the child links of a Collection file was a difficult concurrency problem. We implemented a queue based approach to batch updates into a reasonable size rather than making individual writes but this required additional infrastructure to manage.
  2. Removal of child links was inefficient. Link objects are in an array and required iteration when an Item link needed to be removed.
  3. Parsing large collection files was inefficient. The size of our Collection files grew quite large considering that some of our collections contain > 50M items and required parsing the entire collection file rather than supporting streaming or paging.

For most of our projects it was apparent that managing our Items in a data store that supported concurrency and atomicity would be an easier approach. In work with NASA, AWS and Planetary Computer we use stac-fastapi with a pgstac backend. We’ve tried to provide IAC templates for deploying these packages with simple, intelligent defaults with the eoAPI library that we maintain.

In the case that there is an absolute requirement for static Collection and Item files you could periodically export a representation from your data store. But in practice for large archives of data we’ve found that the STAC API experience is better than working with large numbers of static objects.

Out of curiosity, what is the data for which you are generating large STAC Collections and where is it stored? For datasets managed in S3 we have a fairly basic infrastructure package we often use called stactools-pipelines to manage creating and ingesting new STAC Items to an API as data is published to a bucket. We haven’t been maintaining it closely but let me know if it might be helpful for you.

1 Like