# Best way to wrap scipy.correlate in time

Hi,

I hope I am using this forum in a correct way.
I am struggling with a thing that should be simple. Calculate a correlation!
But I have >300Gb of data and this gets very tricky.

I have a big matrix `A` that has the following dimensions `(time,kx,ky) = (400,4096,4096)`. The first thing I would like to do is to calculate an autocorrelation in time for each `kx` and `ky`, but I would also like to be able to do the same for cross-correlation, as I actually have many matrices like `A` that I would like to correlate with.

To give an example, I will create an arbitrary smaller matrix `A` that is basically a cosine in time adding a noise that depends on the magnitude of `k`. In this example, the fake dataset will be loaded in the memory, but in my real-world example I am loading the datasets from netcdf files using `xr.open_dataset`.

``````import xarray as xr
import numpy as np

time = xr.DataArray(np.arange(200), dims=["time"])
kx = xr.DataArray(np.arange(1000), dims=["kx"])
ky = xr.DataArray(np.arange(1000), dims=["ky"])

K = np.abs(kx + 1j*ky)

A = (
(np.cos(time)*np.exp(-time/50))+
((np.random.randn(time.size)*xr.ones_like(time)*K/2e3))
).assign_coords(time=time, kx=kx, ky=ky)
``````

Anyone has any idea on what would be the fastest way to wrap `scipy.signal.correlate` in a way that I could give any two matrices like `A` and return a matrix `C(lag,kx,ky)` without loading everything in the memory all at once?

The fastest way I could do was to select and load each `kx` and then loop over `ky` selecting each time series, running `scipy.signal.correlate` and then saving into a `numpy.array` that later becomes my `xarray.DataArray` object `C(lag,kx,ky)`.

1 Like

See if xskillscore has something you can use.

How is your data stored on disk?

1 Like

I will take a look. By now, each `A` is stored as a single `nc` file.

1 Like

xarray.corr or xskillscore.pearson_r

4 Likes