# .to\_zarr(..., compute=False) gives zeros not NaNs

**URL:** <https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388>\
**Category:** Uncategorized\
**Created:** [September 5, 2025, 7:25pm UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388 "2025-09-05T19:25:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Len](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/len/32/2954_2.png) [@Len](https://discourse.pangeo.io/u/Len)\
**Post date:** [September 5, 2025, 7:25pm UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388/1 "2025-09-05T19:25:07Z")

</div>

I was hoping that \_FillValue would be used if no chunks have been written yet. For example,

```python
import dask.array as da
import numpy as np
import xarray as xr

ex = xr.Dataset({"data": xr.DataArray(da.full((10, 10), fill_value=np.nan))})

ex.to_zarr("~/tmp/test.zarr", mode="w", compute=False)
rt = xr.open_zarr("~/tmp/test.zarr").compute()
assert rt["data"].isnull().all()

```

raises, when I am actually hoping they would be nan by default. Any idea how to control which value is loaded from empty chunks? I’d like to default this value to the no data value or \_FillValue.

---

<div class="post-metadata">

**Author:** ![Len](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/len/32/2954_2.png) [@Len](https://discourse.pangeo.io/u/Len)\
**Post date:** [September 5, 2025, 7:25pm UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388/2 "2025-09-05T19:25:39Z")

</div>

It looks like with zarr format 3, we can accomplish this with:

```python
ex = xr.Dataset({"data": xr.DataArray(da.full((10, 10), fill_value=np.nan))})
ex['data'].encoding = {"fill_value": np.nan}
ex.to_zarr("~/tmp/test.zarr", mode="w", compute=False, zarr_format=3)
rt = xr.open_zarr("~/tmp/test.zarr").compute()
assert rt["data"].isnull().all()

```

---

<div class="post-metadata">

**Author:** ![sotosoul](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/sotosoul/32/2435_2.png) [@sotosoul](https://discourse.pangeo.io/u/sotosoul)\
**Post date:** [September 6, 2025, 7:12pm UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388/3 "2025-09-06T19:12:00Z")

</div>

The encoding params are missing when to\_zarr(). This works on my side, give it a try:

```python
# Dataset with datavar: 'data', fill_value=np.nan, and dtype='float32':
ds = xr.Dataset(...)

ds.to_zarr(
    store=...,
    compute=False,
    zarr_format=3,
    consolidated=False,
    write_empty_chunks=False,
    encoding={
        'data': {'dtype': np.float32, 'fill_value': np.nan}
    }
)

```

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 8, 2025, 12:41pm UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388/4 "2025-09-08T12:41:58Z")

</div>

The question of missing value / fill value is a bit of a can of worms in Zarr. Here is my brain dump.

### Current Status

- Zarr itself formally has no concept of a “mask” or null values for an array. It only has `fill_value`, the value returned when an entire chunk is not found in the store. This is a part of Array metadata.

- Meanwhile, Xarray implemented its own logic around missing data using the `_FillValue` attribute, using concepts from the [NetCDF User Guide](https://docs.unidata.ucar.edu/netcdf-c/current/file_format_specifications.html) and [CF conventions](https://cfconventions.org/cf-conventions/cf-conventions.html#missing-data). Here’s a relevant quote:

I agree this is all extremely confusing. For some deep background, check out this thread.

> <https://github.com/pydata/xarray/issues/5475>
>
> The zarr backend uses the \`fill\_value\` of zarrs \`.zarray\` key as if it would be …the \`\_FillValue\` according to \[CF-Conventions\](http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#missing-data):
> 
> https://github.com/pydata/xarray/blob/1a7b285be676d5404a4140fc86e8756de75ee7ac/xarray/backends/zarr.py#L373
> 
> I think this interpretation of the \`fill\_value\` is wrong and creates problems. Here's why:
> 
> The \[zarr v2 spec\](https://zarr.readthedocs.io/en/stable/spec/v2.html#metadata) is still a little vague, but states that \`fill\_value\` is
> \> A scalar value providing the default value to use for uninitialized portions of the array, or null if no fill\_value is to be used.
> 
> Accordingly this value should be used to fill all areas of a variable which are not backed by a stored chunk with this value. This is also different from what \[CF conventions state\](http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#missing-data) (emphasis mine):
> \> The scalar attribute with the name \`\_FillValue\` and of the same type as its variable is recognized by the netCDF library as the value used to pre-fill disk space allocated to the variable. \*\*This value is considered to be a special value that indicates undefined or missing data\*\*, and is returned when reading values that were not written.
> 
> The difference between the two is, that \`fill\_value\` is \*\*only\*\* a background value, which just isn't stored as a chunk. But \`\_FillValue\` is (possibly) a background value \*\*and\*\* is interpreted as not being valid data. In my opinion, this mix of \`\_FillValue\` and \`missing\_value\` could be considered a defect in the CF-Conventions, but probably that's far to late as many depend on this.
> 
> Thinking of an example, when storing a density field (i.e. water droplets forming clouds) in a zarr dataset, it might be perfectly valid to set the \`fill\_value\` to \`0\` and then store only chunks in regions of the atmosphere where clouds are actually present. In that case, \`0\` (i.e. no drops) would be a perfectly valid value, which just isn't stored. As most parts of the atmosphere are indeed cloud-free, this may save quite a bunch of storage. Other formats (e.g. \[OpenVDB\](https://www.openvdb.org)) commonly use this trick.
> 
> \---
> 
> The issue gets worse when looking into the upcoming \[zarr v3 spec\](https://zarr-specs.readthedocs.io/en/core-protocol-v3.0-dev/protocol/core/v3.0.html#array-metadata) where \`fill\_value\` is described as:
> \> Provides an element value to use for uninitialised portions of the Zarr array.
> \> 
> \> If the data type of the Zarr array is Boolean then the value must be the literal \`false\` or \`true\`. If the data type is one of the integer data types defined in this specification, then the value must be a number with no fraction or exponent part and must be within the range of the data type.
> \> 
> \> For any data type, if the \`fill\_value\` is the literal \`null\` then the fill value is undefined and the implementation may use any arbitrary value that is consistent with the data type as the fill value.
> \> 
> \> \[...\]
> 
> Thus for boolean arrays, if the \`fill\_value\` would be interpreted as a missing value indicator, only (missing, \`True\`) or (\`False\`, missing) arrays could be represented. A (\`False\`, \`True\`) array would not be possible. The issue applies similarly for integer types as well.

Also relevant:

> <https://github.com/pydata/xarray/issues/10646>
>
> \### What happened?
> 
> With Zarr v2, Xarray conflated \`\_FillValue\` and \`fill\_value\`… (https://github.com/pydata/xarray/issues/5475), so unwritten data in a Zarr file is always decoded as NaN.
> 
> In Zarr v3, this changed. Now, it appears that unwritten data in a Zarr file uses the default Zarr \`fill\_value=0\`.
> 
> In practice, this means that is Zarr metadata is written to disk (with \`compute=False\`) but no values are written, the data will be decoded by xarray as all zeros instead of all NaNs. This is perhaps unavoidable for integer data (there is no integer NaN), but for floats, we should default to a user controllable \`fill\_value\` of NaN, which is clearly an invalid value.
> 
> To reproduce:
> \`\`\`python
> import xarray
> import numpy as np
> 
> ds = xarray.Dataset({'foo': ('x', np.ones(3))}).chunk()
> 
> path = '/tmp/foo5.zarr'
> ds.to\_zarr(path, compute=False)
> 
> \# all NaN with zarr v2, all zeros with zarr v3
> print(xarray.open\_zarr(path).compute())
> \`\`\`
> 
> Possibly related: https://github.com/pydata/xarray/issues/10633, https://github.com/pydata/xarray/issues/10269

### Where to go from here

We are currently relying on a pretty fragile and poorly documented set of assumptions around how to handle missing data in Xarray / Zarr. We should overcome this formalizing the concept of missing data / null values at the Zarr level. That way, Xarray could basically skip its own encoding / decoding of fill values and rely on Zarr to do it.

The place to start would probably be to promote `_FillValue` to a “registered attribute” of Zarr and implement mask encoding / decoding within Zarr itself. See [https://github.com/zarr-developers/zeps/pull/67#issuecomment-3220214413](https://github.com/zarr-developers/zeps/pull/67#issuecomment-3220214413) for some discussion of that.

Going further, we could attempt to copy some of the good ideas in other formats like Arrow / Parquet, which actually store a null-value mask as an independent buffer alongside the array values.

---

<div class="post-metadata">

**Author:** ![kirill.kzb](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/kirill.kzb/32/1276_2.png) [@kirill.kzb](https://discourse.pangeo.io/u/kirill.kzb)\
**Post date:** [September 9, 2025, 3:49am UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388/5 "2025-09-09T03:49:05Z")

</div>

I have always struggled with `fill_value` / `nodata` concepts in `odc.{geo,stac}`. I think it’s due to the fact that it tries to be two similar but different things: sentinel value for `NaN`, when working with integer or fixed width float types and just “fill value” for pixels that do not overlap with the source imagery. We had to add fairly involved `dtype` dependent logic for figuring out what value to default to when not supplied by the user or the underlying source data.

Most of the time “fill value” is also a “sentinel for `NaN`", but not always. Take 8 bit RGB visual sources as an example, all black pixel `0,0,0` could be a reasonable fill value for blocks that were not recorded, but it can also appear as part of the valid data, so can not be relied on to build “valid pixel mask”.

To me it feels wrong to use anything but `NaN` as a fill value for floating point data, but some data sources do that, while also having `NaN`s in the data… In fact even defining that on `float` data feels wrong, as it’s part of IEEE float spec already, so one should assume that `NaN` might be present.

Somewhat off topic: how does zarr handle `NaN` values in the metadata sections, given that it’s JSON under the hood, and it ain’t valid for JSON to have `NaN` or `inf`?

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 9, 2025, 1:12pm UTC](https://discourse.pangeo.io/t/to-zarr-compute-false-gives-zeros-not-nans/5388/6 "2025-09-09T13:12:36Z")

</div>

Perhaps we are going off topic, but Kirill, I completely agree with your message. The meaning of “missing data” is very context-dependent, and different applications need to handle it in very different ways. For floating-point data in Zarr, it is always possible to just put NaNs directly in the data, without any special metadata `fill_value`, `_FillValue`, etc. But it’s up to the user to interpret what this means. For integer types, the lack of a NaN makes this much harder.

> [@kirill.kzb](#):
>
> ow does zarr handle `NaN` values in the metadata sections, given that it’s JSON under the hood, and it ain’t valid for JSON to have `NaN` or `inf`?

Good question. Zarr uses the string “Nan”: [Zarr core specification — Zarr specs documentation](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#example)

In Python, it does this by calling `json.dumps(…, allow\_nan=True). (See [json — JSON encoder and decoder — Python 3.13.7 documentation](https://docs.python.org/3/library/json.html#json.dump))