Interesting blog post on a new google package. Looks like a lot of (potential) overlap with our community.
2 Likes
Tensorstore is a great project that we have been following here for a while! Tensorstore reads and writes Zarr data, so it has a lot of interoperability with the Pangeo ecosystem. In terms of stack, I see Tensorstore as a possible alternative to zarr-python and fsspec for Zarr I/O.
Some interesting things to try with Tensorstore if anyone wants to play around:
- Compare read performance on Tensorstore vs. zarr-python + fsspec on big cloud datsets (e.g. Multi-Scale Ultra High Resolution (MUR) Sea Surface Temperature (SST) - Registry of Open Data on AWS)
- Try to create an Xarray backend for Tensorstore
5 Likes
I’m curious if anyone is working on either of @rabernat’s suggested tasks? (benchmarking and creating an xarray backend)? They’d be super useful!
1 Like
I think @shoyer is working on the backend.
1 Like
Yes, I have a TensorStore backend (of sorts) for xarray working, which we’ve been for loading data for ML training.
It’s basically ready for release as a new mini open source project, but I got a little side tracked. I’ll see if I can finish that up over the next week or so and will update this thread!
5 Likes
Sounds fantastic - thank you!
1 Like
OK, my TensorStore adapter is up on GitHub now:
5 Likes