# How to save out-of-memory Xarray file to .csv

**URL:** <https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579>\
**Category:** Data\
**Created:** [July 20, 2023, 7:05pm UTC](https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579 "2023-07-20T19:05:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![BEStecher](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/bestecher/32/2335_2.png) [@BEStecher](https://discourse.pangeo.io/u/BEStecher)\
**Post date:** [July 20, 2023, 7:05pm UTC](https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579/1 "2023-07-20T19:05:03Z")

</div>

I am trying to save a large Xarray dataset to csv, however, I have been struggling with memory issues and dataset formatting. Since my data is quite large, I have been utilizing Dask and opening up my netCDF files using:

```auto
xarray_data_ssp126 = xr.open_mfdataset('filename*.nc', combine='by_coords', parallel=True)

```

I do not want to use every single point in the data above, so below I am importing my desired lat/lon and converting it from .csv (gem\_df) to xarray (gem):

```auto
gem = gem_df.to_xarray()

gem = gem.assign_coords(lat = ("lat", gem_df.lat), lon = ("lon", gem_df.lon))

```

I combined the gem dataset and the xarray\_data\_ssp126 dataset using interpolation with nearest neighbor:

```auto
data = xarray_data_ssp126.interp(lat = gem.lat, lon = gem.lon, method = 'nearest')

```

I then wanted to combine the ‘data’ dataset with gem to have some of my other data variables back.

```auto
data = data.combine_first(gem)

data

```

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/5/57d69001241306dc9723e5ab56d1897ac557fd67.png)

Which leaves me with the exact data I want (yay!), however, I have been struggling to find a way to export this data due to memory issues. I have read that you have previously suggested exporting to zarr, which did not work for me unfortunately. I tried to minimize the data by going from all United States data to just state level data, which did not work either. The data is daily and I am trying to save it in chunks of 25 years.

I have been able to do this previously when ignoring the coordinates (not sure if this is the right terminology) by using set\_coords((“lat”, “lon”)) instead of the assign\_coords listed above and instead of interpolating using select with the nearest neighbor method. By ignoring the coordinates I mean that when I did not assign the coordinates for ‘gem’, my data variables just said “index” instead of time, lat, lon.

I was wondering if you have any advice on how to go about saving these to csv?

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [July 20, 2023, 7:31pm UTC](https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579/2 "2023-07-20T19:31:17Z")

</div>

Thanks for posting this interesting question.

> [@BEStecher](#):
>
> Which leaves me with the exact data I want (yay!)

I’m not convinced that this is actually the exact data you want. Based on your description, it looks like you want a dataset with 1121 different points in it (and 31411 timesteps); i.e. a 2D array. However, here you have a 3D array (31411 x 1121 x 1121); a subsample of your original datacube. This is 1121 times bigger than what you want, which could be why you are running out of memory.

You should probably be using Xarray’s “vectorized indexing” here:

> **[Indexing and selecting data](https://docs.xarray.dev/en/stable/user-guide/indexing.html#vectorized-indexing)**
>
> Xarray offers extremely flexible indexing routines that combine the best features of NumPy and pandas for data selection. The most basic way to access elements of a DataArray object is to use Pytho...

For example, something like

```python
lat_index = xr.DataArray(gem_df.lat, dims="point")
lon_index = xr.DataArray(gem_df.lon, dims="point")
data = xarray_data_ssp126.sel(lon=lon_index, lat=lat_index, method="nearest")

```

I’m still not certain that will work properly with Dask. But at least it should be the right data structure.

Try this out and report back here!

---

<div class="post-metadata">

**Author:** ![BEStecher](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/bestecher/32/2335_2.png) [@BEStecher](https://discourse.pangeo.io/u/BEStecher)\
**Post date:** [July 20, 2023, 9:42pm UTC](https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579/3 "2023-07-20T21:42:45Z")

</div>

Continuing the discussion from [How to save out-of-memory Xarray file to .csv](https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579/2):

Hi!

First of all, thank you so much for replying!

I was able to get the code to work as you had suggested, and to clarify, I am looking for a dataframe that has the time, the surface wind speed value, and the coordinates (lat, lon). So yes, a 2d array I think! I was able to have the code work by implementing what you suggested:

```auto
lat_index = xr.DataArray(gem_df.lat, dims="point")
lon_index = xr.DataArray(gem_df.lon, dims="point")
data = xarray_data_ssp126.interp(lon=lon_index, lat=lat_index, method="nearest")
data = data.combine_first(gem)

```

Which results with:

 ![Screenshot 2023-07-20 171055](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/a/a9b16d8fcc9ae547889d20bc23d63fef26d19032.png)

I was wondering why not just replace ‘points’ with ‘index’ since they have the same number (and if it is okay to have that many dimensions)?

I am still facing the same issues regarding saving, and even trying to swtich it to a dask dataframe. I am trying to save in chunks of 25 years, so instead of 31411 time steps I just want 9131, which does not work either.

Unable to save to csv [more like unable to make it past to\_dataframe()]  
`MemoryError: Unable to allocate 3.53 GiB for an array with shape (3652, 360, 720) and data type float32`

and unable to save to dask dataframe:  
`MemoryError: Unable to allocate 102. PiB for an array with shape (14419187630906411,) and data type datetime64[ns]`

---

<div class="post-metadata">

**Author:** ![BEStecher](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/bestecher/32/2335_2.png) [@BEStecher](https://discourse.pangeo.io/u/BEStecher)\
**Post date:** [July 26, 2023, 1:50pm UTC](https://discourse.pangeo.io/t/how-to-save-out-of-memory-xarray-file-to-csv/3579/4 "2023-07-26T13:50:55Z")

</div>

Hi all! I ended up being able to transfer my data to dask after using the .sel method instead of the .interp method. Once in dask, I was able to save to .csv. Not sure why this only worked for me using .sel instead of .interp, but I am happy it did 🙂