Vector data cubes

martinfleis · November 14, 2022, 10:28pm

I have started the development in GitHub - martinfleis/xvec: Vector data cubes for Xarray. Everyone is welcome to join the effort. I’ll try to outline the roadmap in the following days.

benbovy · November 15, 2022, 10:04am

I see Xoak as a package that only provides a few “basic” Xarray indexes based on tree structures (BTree, KDTree, BallTree, etc.). Those may be useful for geospatial operations but also for other purposes, i.e., I don’t think Xoak indexes should be CRS-aware.

My feelings about the Xarray geospatial ecosystem is that the parallel development of multiple projects each with their own, well-defined scope will better help growing the ecosystem (all-things-geospatial is a wide domain! vectors, rasters, etc.).

Hopefully, at some point it will be possible to bring everything together nicely. For example, provide in geoxarray a high-level, CRS-aware GeoIndex “meta-index” that would internally reuse one of the xoak, xvec, rioxarray, etc. indexes depending on the case.

rabernat · February 8, 2023, 1:52pm

This discussion seems to be continuing on github:

github.com/martinfleis/xvec

saving and reading vector data cubes

opened 09:30PM - 07 Feb 23 UTC

martinfleis

At the moment, we have a decent set of functionality to work with in-memory vect…or data cubes. What we do not have is a way of saving them to a disk and reading back again, apart from using `pickle` which is not very interoperable. We should find a way of saving to a file format that allows us interoperability with @edzer's `stars` in RSpatial. @edzer, you were mentioning that NetCDF is able to handle VDC. Does `stars` implements that? Can we create a data cube indexed by geometry and save it via GDAL to NetCDF? I tried only very briefly to play with the `write_stars` function to no avail[^1]. The spec for NetCDF way of storing geometries is [here](http://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#geometries) but as someone who never worked with the file format, it is not super clear to me how it all works together. Over in Python, there is some work @dcherian posted on conversion of shapely geometries to CF geometries and back (https://cf-xarray.readthedocs.io/en/latest/generated/cf_xarray.shapely_to_cf.html) [^2]. There is also https://github.com/twhiteaker/cfgeom which may potentially be used (although it is only a reference implementation that is not maintained). So it seems that NetCDF should be able to store vector data cubes and we may just need to figure out some wrappers for a convenient IO from xvec-backed xarray objects. But I'd welcome some help or at least a guidance from someone more familiar with CF conventions. The other option seems to be Zarr as discussed [here](https://github.com/zarr-developers/geozarr-spec/issues/3) but that is at this stage only an idea and (Geo)Zarr spec is not ready as of now. The last option is to convert the cube to a long-form dataframe and save it as a GeoParquet but that kind of breaks the point of having a cube in the first place. [^1]: Getting `Error in !all.equal(match(xydims, names(d)), 1:2) : invalid argument type` when trying to use `write_stars(st, 'test.nc')` on the example cube from https://r-spatial.github.io/stars/articles/stars1.html#vector-data-cube-example. [^2]: only points are implemented so far but that may change

allixender · March 8, 2023, 8:53am

Xvec seems to have moved to xarray-contrib, maybe #26 is still the one?

Anyway, I see great potential, e.g. to experiment with Xvec and Xoak in the space of DGGS. Unfortunately, we still haven’t moved much. We are currently, trying to bring DGGRID onto conda-forge.

allixender · July 3, 2023, 8:15am

DGGRID is on conda-forge and the lead developer is restructuring the internal API. It should become easier now to develop bindings to other languages such as Python and Julia. We would then want to explore various interop/indexing/analysis schemes with xarray, such as xoak and xvec, and then also moving between unstructured grids and dggs grids. There will be a code sprint in November at the Vienna BIDS 2023 conference. Would be great to have some Pangeo core devs there

martinfleis · July 3, 2023, 8:33am

I am considering going to BIDS, this seems to be another reason to go. Not that I know what dggs grids are but I am happy to learn how Xvec can support that :).

allixender · September 4, 2023, 2:31pm

@martinfleis Discrete Global Grid Systems (DGGS) use with Pangeo - #39 by allixender

rabernat · September 4, 2023, 7:29pm

I will be going to BiDS and am excited to hack on vector data cubes.

martinfleis · September 19, 2023, 9:02am

I will not be at BiDS due to teaching duties, but I’ll try to be available online for some sprinting.

Topic		Replies	Views
What's Next - Software - Geospatial Bridge	1	191	December 12, 2023
Xarray Accessor for Georeferenced Data Comparisons	6	373	March 23, 2023
Introducing pyvista-xarray for 3D visualization	3	1459	July 11, 2022
Tables, (x)arrays, and rasters¶	18	2650	November 15, 2022
Xarray for raster data (DEMs) with inconsistent spatial extent Data	10	2626	January 6, 2024

Vector data cubes

Related Topics