EDIT: I realised after creating this that it probably isn’t the best channel - I have created #Zarr > Bindings from Zarrs to other languages (Julia, R etc.) in the Zulip where people are chiming in. Let’s continue discussion there (if you already have an account, otherwise here).
I was thinking about something that @TomNicholas mentioned about the Rust implementation of Zarr having a bunch of optimisations that aren’t accounted for in implementations in other languages.
I was wondering whether people have or are currently exploring bindings of the Zarr Rust implementation “zarrs” into other languages? Perhaps a Python implementation would be nice to use to compare against zarr-python
I think as far as inspiration goes GitHub - kylebarron/arro3: A minimal Python library for Apache Arrow, connecting to the Rust Arrow crate (providing Python → Rust bindings for Arrow) might be a good source - I think in a similar spirit keeping functionality close to the zarrs package will allow for a low maintenance burden and better coherence.
I have a good friend who’s been itching a bit to do some climate related Rust open source contributions, and thought this would be an impactful option he might be interested in (personally, I’m also keen to also check out more Rust and what bindings into Python/other languages look like).
(I’m not super familiar with Zarr development - this may have already been discussed, keen to be brought more up to speed!)
1 Like
My friend just pointed out that Zarrs already has some bindings zarrs - Zarr V2/V3 for Rust · GitHub ! Though currently only to Python and C++
I agree zarrs seems the most promising library to rally around for cross-lang support.
I don’t have much Rust or even Zarr/python experience, I’ve been contributing to the R package gdalraster (in GDAL I’m much more proficient and there are a number of opportunities to improve in both Python and R). I consider GDAL (C++) to be an analogous lower level library for Zarr support, analagous to zarrs.
We’re working on multdim features for gdalraster Report structure and content of a multidimensional dataset — mdim_info • gdalraster and I’m working on a GDAL backend for xarray GitHub - mdsumner/gdx (rioxarray is bound to the 2D raster model only, so I consider that a false step for Zarr and is not going to be able to extend to true multidim). GDAL is not exactly related to your topic, but because of the depth of Zarr support it has and the really low usage of the multidim mode I think it’s relevant to share. Happy to see discussion! I’ll point others to this.
I did actually start a R wrapper for zarrs, but I’m reluctant to point at it because I’m so out of my depth. I’ve had some good success with Rust and R since but I’m only learning slowly, the support is quite good via GitHub - extendr/extendr: R extension library for rust designed to be familiar to R users. and GitHub - yutannihilation/savvy: A simple R extension interface using Rust
1 Like