Using xhistogram to bin measurements at particular stations

rabernat · March 30, 2022, 3:16pm

Ok that is much more clear. Thanks for taking the time to write it up.

You’re correct that it is not possible today with xhistogram. The reason is that, as currently implemented, xhistogram relies heavily on the fact that it is easy to just sum up the bin counts from each block of data to reach the total for each. Sum is commutative and associative, so it is trivial to parallelize (and most of the code in xhistogram is about making things play well with dask).

FWIW, the algorithm itself is here and is not that complicated to read.

github.com

xgcm/xhistogram/blob/master/xhistogram/core.py

"""
Numpy API for xhistogram.
"""


import dask
import numpy as np
from functools import reduce
from collections.abc import Iterable
from numpy import (
    searchsorted,
    bincount,
    reshape,
    ravel_multi_index,
    concatenate,
    broadcast_arrays,
)

# range is a keyword so save the builtin so they can use it.
_range = range

This file has been truncated. show original

Dan_Jones:

Even better would be a command that lets me simply do the following. (I’m totally making this code up, no idea if it makes sense…)
A = da.groupby(['lon_bins', 'lat_bins']).mode()

This is a very long-standing open issue in xarray:

github.com/pydata/xarray

Support multi-dimensional grouped operations and group_over

opened 07:42PM - 18 Feb 15 UTC

shoyer

API design topic-groupby

Multi-dimensional grouped operations should be relatively straightforward -- the… main complexity will be writing an N-dimensional concat that doesn't involve repetitively copying data. The idea with `group_over` would be to support groupby operations that act on a single element from each of the given groups, rather than the unique values. For example, `ds.group_over(['lat', 'lon'])` would let you iterate over or apply to 2D slices of `ds`, no matter how many dimensions it has. Roughly speaking (it's a little more complex for the case of non-dimension variables), `ds.group_over(dims)` would get translated into `ds.groupby([d for d in ds.dims if d not in dims])`. Related: #266

I think that the Flox package by @dcherian supports it. Let’s see what Deepak has to say about this.

Topic		Replies	Views
Xarray.dataset.grouby_bins without squishing other dimensions	4	1671	August 10, 2021
Usage of xhistogram compared to np.digitize Science	1	413	April 10, 2021
Python taking statistics over each latitudes and longitudes pair Data	3	580	November 26, 2020
Handling slicing with circular longitude coordinates in xarray Visualization	9	2485	August 2, 2022
Best practices for inferring meaning from xarray Datasets? Science	4	571	September 10, 2021

Using xhistogram to bin measurements at particular stations

Related Topics