Welcome @ags_pangeo!
I think the formula that our community has converged around for optimal performance and flexibility is Zarr. Combined with XPublish’s EDR plugin, this gives you a great architecture to achieve your needs. In fact, @jhamman and Alex Kerney recently gave a presentation on this exact subject at the Pangeo showcase.
If you’re interested in a hosted solution, rather than something you have to build and manage yourself, our company Earthmover offers a subscription-based managed service using this type of architecture.