A quick and dirty Xpublish plugin for accessing Zarr stores over HTTPS or S3:
class WebDatasetPlugin(xpublish.Plugin):
name: str = "web-dataset-provider"
@xpublish.hookimpl
def get_datasets(self):
return ["zarr+https", "zarr+s3"]
@xpublish.hookimpl
def get_datatree(self, dataset_id: str, group: str):
if dataset_id == "zarr+https":
ds = xr.open_zarr(f"https://{group}")
return xr.DataTree(dataset=ds)
if dataset_id == "zarr+s3":
mapper = fsspec.get_mapper(f"s3://{group}")
ds = xr.open_zarr(mapper, consolidated=True)
return xr.DataTree(dataset=ds)
In Xpublish parlance, it treats the format and protocol as the dataset_id and the group is then the rest of the URL.
In use for HTTPS:
Or S3:
I whipped up a quick stats plugin to give a mean and started a request (http://localhost:9005/datasets/zarr+s3/groups/mur-sst/zarr-v1/mean/analysed_sst) without thinking too much about it…
class StatPlugin(xpublish.Plugin):
name: str = "stat-plugin"
@xpublish.hookimpl
def dataset_router(self, deps: xpublish.Dependencies):
from fastapi import Depends
from fastapi.responses import JSONResponse
from fastapi.routing import APIRouter
router = APIRouter(prefix="", tags=["stat"])
@router.get("/groups/{group_path:path}/mean/{var}")
def mean(
dataset=Depends(deps.dataset),
var: str,
) -> JSONResponse:
"""Returns the mean of a variable in the dataset."""
means = dataset[var].mean()
return JSONResponse(means.to_dict())
return router
Xpublish has happily streamed over 200 GB while calculating the mean before I decided it was time for bed and killed it.
Dove into async jobs before realising you had mentioned sync HTTP responses
So some sort of async job submission type API is probably gonna be the way. For existing standards like OpenEO or OGC Processes (more REST-ful replacement for WCPS), the question becomes 'how complex of a query are you looking to support?`
Both of them have a form of job submission then digital thumb twiddling while waiting for a result.
OGC Processes is on the simpler side with one-off processes, and is more of a standard for the URL patterns than how you tell it what data to work on. So in some ways it would be easier to adapt to querying any Zarr store by including an input like zarr_url for all processes. pygeoapi has some support for the standard.
OpenEO allows chaining of processes together, running them synchronously when small enough, as batch jobs when they are bigger, or connecting them to a service like WMS for visualization (a server doesn’t have to support all of them, and there is a built in way to tell users to ask for less). It has from what I can tell a better set of existing API clients since it’s a more fully defined standard. From what I’ve seen it’s generally more focused on producing raster results, but it can get to JSON timeseries as well. I think that means it will take a little more creativity to figure out how to fit any HTTP accessible Zarr URL into how you specify a data cube, though I think collection_property could be molded into the right shape by doing a similar virtual collections/dataset trick like I did in Xpublish.
I’d lean towards the openEO API. It nicely is already structured to scale from sync responses > async jobs > web services, auth, and has an ‘you’ve asked for too much’ error with a spread of clients that understand that. WCPS and it’s successor OGC Processing are both for async jobs.
openEO API profiles
It is structured around named STAC items, but I think there are two ways to solve that.
- Virtual collections using/abusing the
collection_property could be molded into the right shape like I did in Xpublish.
- For more flexibility, add some state and
POST/PUT datasets as STAC collections on demand. You could probably accept some form of xr.open_dataset() kwargs. This also could give the API a easier ID to cache responses against, and potentially an easier entry point than ‘learn how to get creative with collection_property’. Though getting creative with collection_property could also be having collection_property consume open_dataset kwargs as well.
Maybe the move would be to support both, but encourage regular uses to register (give them higher limits), and to create their commonly used collections.
I now really want to build an openEO pluggable ecosystem on top of Xpublish. Anyone want to contract my team to do so since I really can’t justify that with what we currently have in our pipeline?
My team on the NASA Office of Data Science and Informatics (ODSI) Data Systems Evolution (DSE) program is considering building an HTTP API for statistics generation from Zarr stores and we could use the communities input.
