# Xarray trouble decoding NetCDF with compressed integers

**URL:** <https://discourse.pangeo.io/t/xarray-trouble-decoding-netcdf-with-compressed-integers/4646>\
**Category:** Data\
**Created:** [October 30, 2024, 7:27pm UTC](https://discourse.pangeo.io/t/xarray-trouble-decoding-netcdf-with-compressed-integers/4646 "2024-10-30T19:27:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jalder](https://avatars.discourse-cdn.com/v4/letter/j/f05b48/32.png) [@jalder](https://discourse.pangeo.io/u/jalder)\
**Post date:** [October 30, 2024, 7:27pm UTC](https://discourse.pangeo.io/t/xarray-trouble-decoding-netcdf-with-compressed-integers/4646/1 "2024-10-30T19:27:42Z")

</div>

Hi all, I just came across a curious case where xarray is throwing this error on a compressed integer stored in a NetCDF4:

RuntimeWarning: overflow encountered in scalar absolute  
vlim = max(abs(vmin - center), abs(vmax - center))

Here is the encoding as seen by xarray:  
{‘dtype’: dtype(‘int16’),  
‘zlib’: True,  
‘szip’: False,  
‘zstd’: False,  
‘bzip2’: False,  
‘blosc’: False,  
‘shuffle’: False,  
‘complevel’: 1,  
‘fletcher32’: False,  
‘contiguous’: False,  
‘chunksizes’: (151, 79, 118),  
‘preferred\_chunks’: {‘time’: 151, ‘lat’: 79, ‘lon’: 118},  
‘original\_shape’: (151, 474, 944),  
‘missing\_value’: -999,  
‘\_FillValue’: -999}

Xarray doesn’t seem to be correctly decoding and applying the missing value -999. As a side note, Panoply does. Here the ncdump shown in Panoply:

short CDD(time=151, lat=474, lon=944);  
:missing\_value = -999S; // short  
:\_FillValue = -999S; // short  
:long\_name = “Consecutive Dry Days”;  
:units = “days”;  
:\_ChunkSizes = 151U, 79U, 118U; // uint

xarray is loading the variable as an int64, but since isn’t catching the missing\_value correctly the missing grid cells (ie ocean and lake bodies in these data) are loading as -9223372036854775808, which is messing up spatial averaging by not being represented as a NaN. My other compressed variables that have a scale\_factor and add\_offset are correctly being decoded into floats with missing\_value NaNs.

Is the missing\_value = -999S as a short forcing xarray to load the variable as an integer rather than a float with NaNs?

This is publicly released data, so I can’t change the source NetCDF files, but maybe I can write a xarray preprocess function to correct the typing.

Thanks for any suggestions.

---

<div class="post-metadata">

**Author:** ![ThomasMGeo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/thomasmgeo/32/1722_2.png) [@ThomasMGeo](https://discourse.pangeo.io/u/ThomasMGeo)\
**Post date:** [November 2, 2024, 2:16am UTC](https://discourse.pangeo.io/t/xarray-trouble-decoding-netcdf-with-compressed-integers/4646/2 "2024-11-02T02:16:08Z")

</div>

Can you share some example data?

---

<div class="post-metadata">

**Author:** ![jalder](https://avatars.discourse-cdn.com/v4/letter/j/f05b48/32.png) [@jalder](https://discourse.pangeo.io/u/jalder)\
**Post date:** [November 4, 2024, 4:59pm UTC](https://discourse.pangeo.io/t/xarray-trouble-decoding-netcdf-with-compressed-integers/4646/3 "2024-11-04T16:59:48Z")

</div>

Sure, the data release is from [here](https://www.sciencebase.gov/catalog/item/65cd1ff2d34ef4b119cb3d07). There are many files, but the CMIP6-LOCA2\_Thresholds\_AllModels\_grid\_R3in.tar.gz file is the smallest and demonstrates the compressed integer issue.

```auto
import numpy as np
import xarray as xr

sample_file = '/Volumes/head4/published_projects/ScienceBase_Alder_2024_CMIP6-LOCA2_Thresholds/CMIP6-LOCA2_Thresholds_AllModels_grid/R3in/CMIP6-LOCA2_Thresholds_R3in_ACCESS-CM2.ssp245.r1i1p1f1_1950-2100_16thdeg_grid.nc'
ds = xr.open_dataset(sample_file, decode_times=False)

ds.R3in.isel(time=0).plot()

print(ds.R3in[0,0,0].data)

```

Which prints -9223372036854775808, being the xarray processed value for water bodies in these data. The missing\_value is -999S, but xarray is still loading it as int64 rather than a double or a float that can support NaNs.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 5, 2024, 1:43am UTC](https://discourse.pangeo.io/t/xarray-trouble-decoding-netcdf-with-compressed-integers/4646/4 "2024-11-05T01:43:31Z")

</div>

Thanks for sharing! Hopefully someone can look into this. Sounds like an issue for the [Xarray issue tracker](https://github.com/pydata/xarray/issues).

Perhaps one unintentional takeway from this thread is that a `.tar.gz` file containing NetCDFs is not a particularly accessible or easy way to share and distribute data! 😆 I wanted to investigate this myself, but I gave up when I saw the amount of friction ahead of me to get to an actual Xarray dataset. (Lazy I know, but that’s probably typical.)