# Processing large (too large for memory) xarray datasets, and writing to netcdf

**URL:** <https://discourse.pangeo.io/t/processing-large-too-large-for-memory-xarray-datasets-and-writing-to-netcdf/1724>\
**Category:** Science\
**Created:** [September 1, 2021, 8:32am UTC](https://discourse.pangeo.io/t/processing-large-too-large-for-memory-xarray-datasets-and-writing-to-netcdf/1724 "2021-09-01T08:32:08Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 3, 2021, 12:25pm UTC](https://discourse.pangeo.io/t/processing-large-too-large-for-memory-xarray-datasets-and-writing-to-netcdf/1724/8 "2021-09-03T12:25:50Z")

</div>

An alternative approach, if you know the total number of timesteps in advance, would be to initialize the whole Zarr store at the beginning with a lazy “template” dataset, i.e.

```python
ds_template.to_zarr(tempdir, compute=False)

```

and then use the `region` argument to write, i.e. something like

```python
for j in range(ntimesteps):
    dd.to_zarr(tempdir, region={'time': slice(j, j+1)}

```

This might be faster because it won’t actually _open_ the Zarr store with Xarray each time. That’s what I think is happening with `append`.

---

_[View the full topic](https://discourse.pangeo.io/t/processing-large-too-large-for-memory-xarray-datasets-and-writing-to-netcdf/1724)._