# How to identify the array addresses in NetCDF/HDF files (for fsspec-reference-maker)?

**URL:** <https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830>\
**Category:** Data\
**Created:** [October 7, 2021, 1:40pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830 "2021-10-07T13:40:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![agstephens](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/agstephens/32/516_2.png) [@agstephens](https://discourse.pangeo.io/u/agstephens)\
**Post date:** [October 7, 2021, 1:40pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830/1 "2021-10-07T13:40:05Z")

</div>

Hi @martindurant,

I watched your excellent video about the ReferenceFileSystem and took a look at:

> **[GitHub - intake/fsspec-reference-maker: Functions to make reference...](https://github.com/intake/fsspec-reference-maker)**
>
> Functions to make reference descriptions for ReferenceFileSystem - GitHub - intake/fsspec-reference-maker: Functions to make reference descriptions for ReferenceFileSystem

Do you have any code/docs/tips on how to find out where the arrays (or compressed arrays) are kept inside a NetCDF/HDF file? Can you use the netCDF4/HDF5 libraries to interrogate the internal structure in order to find these?

Thanks, Ag

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [October 7, 2021, 1:56pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830/2 "2021-10-07T13:56:47Z")

</div>

Yes indeed, and this is what the hdf module in that repo exactly does, using h5py. You can run it pretty much like a script (on a pangeo-forge recipe) for individual files and combine them into aggregate datasets too (in the combine module).

---

<div class="post-metadata">

**Author:** ![agstephens](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/agstephens/32/516_2.png) [@agstephens](https://discourse.pangeo.io/u/agstephens)\
**Post date:** [October 7, 2021, 2:28pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830/3 "2021-10-07T14:28:37Z")

</div>

Thanks @martindurant, that’s great.

Can it do all this by only reading the NetCDF Header?

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [October 7, 2021, 3:13pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830/4 "2021-10-07T15:13:05Z")

</div>

Correct, it only reads the metadata (which is generally spread throughout the file - it’s only all in the header if you are lucky).

---

<div class="post-metadata">

**Author:** ![agstephens](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/agstephens/32/516_2.png) [@agstephens](https://discourse.pangeo.io/u/agstephens)\
**Post date:** [October 7, 2021, 3:26pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830/5 "2021-10-07T15:26:12Z")

</div>

Thanks @martindurant

It works first time on a random ERA5 file I tested. Looks really promising.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [October 7, 2021, 3:38pm UTC](https://discourse.pangeo.io/t/how-to-identify-the-array-addresses-in-netcdf-hdf-files-for-fsspec-reference-maker/1830/6 "2021-10-07T15:38:36Z")

</div>

Great! Keep us informed or open an issue at reference-maker with problems or success 🙂

I recommend using `cache_type="first"` when using fsspec to open files to pass to h5py, since _most_ of the metadata tends to be at the start of the file, and other metadata pieces are randomly scattered, so read-ahead doesn’t help.