I have been contributing to an open source AGPL project GitHub - HuskHoard/HuskHoard: An open-source (AGPLv3), Rust-based automated data-tiering and archiving engine for Linux. Transparently offload cold data to S3, HDD, or LTO tape while keeping searchable local stubs. · GitHub and a colleague mentioned there may be a good use case here. It is an open source storage management system that runs on Linux. It archives files based on age to different storage tiers LTO, disk, and Cloud. It leaves a stub in the file system and records the placement in a database, recalling a file is just clicking on the file in the fs. The files are stored as Zstd frames and metadata in the catalog contains the file contents. Data can be recalled in partial files by restoring selected frames. I wrote a piece with some more details here. I hope you find it useful. The Needle in the Haystack Problem: Solving Earth Science Data Gravity — HuskHoard