# Making kerchunk as simple as a toggle?

**URL:** <https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690>\
**Category:** Open Science\
**Created:** [September 21, 2023, 4:31am UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690 "2023-09-21T04:31:05Z")\
**Posts on this page:** 11\
**Page:** 2

<div class="post-metadata">

**Author:** ![Michael\_Sumner](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/michael_sumner/32/1608_2.png) [@Michael\_Sumner](https://discourse.pangeo.io/u/Michael_Sumner)\
**Post date:** [August 13, 2024, 8:42pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/22 "2024-08-13T20:42:30Z")

</div>

This, I think: [hdf4doc/DSpec/html\_FM/DS.pdf at master · HDFGroup/hdf4doc · GitHub](https://github.com/HDFGroup/hdf4doc/blob/master/DSpec/html_FM/DS.pdf)

---

<div class="post-metadata">

**Author:** ![Michael\_Sumner](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/michael_sumner/32/1608_2.png) [@Michael\_Sumner](https://discourse.pangeo.io/u/Michael_Sumner)\
**Post date:** [August 13, 2024, 8:47pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/23 "2024-08-13T20:47:31Z")

</div>

in case examples are useful, these are ones I’m exploring:

this one has matching GeoTIFF, it’s a very simple single-array case:  
[https://data.seaice.uni-bremen.de/amsr2/asi\_daygrid\_swath/s6250/2024/aug/Antarctic/](https://data.seaice.uni-bremen.de/amsr2/asi_daygrid_swath/s6250/2024/aug/Antarctic/)

these are more complicated, with multiple 3D arrays (needs earthdata auth)

> **[Archive - LAADS DAAC](https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD021KM/2024/226/)**
>
> Level-1 and Atmosphere Archive and Distribution System Web Interface

---

<div class="post-metadata">

**Author:** ![rsignell](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rsignell/32/447_2.png) [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Post date:** [August 19, 2024, 9:07pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/24 "2024-08-19T21:07:44Z")

</div>

At the ESIP 2024 Summer Meeting, @jgallagher59701 from OPeNDAP gave a talk that included discussion about some of the the crazy stuff that goes on inside those NASA HDF4 files. Check out his talk [VirtualiZarr and DMR++](https://zenodo.org/records/13176038).

---

<div class="post-metadata">

**Author:** ![jgallagher59701](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jgallagher59701/32/2781_2.png) [@jgallagher59701](https://discourse.pangeo.io/u/jgallagher59701)\
**Post date:** [August 19, 2024, 9:38pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/25 "2024-08-19T21:38:13Z")

</div>

Thanks Rich! I should point out that the work was all done by Ayush Nag.  
If anyone has questions about reading/decoding HDF4, I’ll try and help.

---

<div class="post-metadata">

**Author:** ![Michael\_Sumner](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/michael_sumner/32/1608_2.png) [@Michael\_Sumner](https://discourse.pangeo.io/u/Michael_Sumner)\
**Post date:** [August 20, 2024, 4:01am UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/26 "2024-08-20T04:01:28Z")

</div>

ah sweet, thanks very much 🙏

---

<div class="post-metadata">

**Author:** ![keewis](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/keewis/32/1106_2.png) [@keewis](https://discourse.pangeo.io/u/keewis)\
**Post date:** [August 20, 2024, 10:35am UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/27 "2024-08-20T10:35:12Z")

</div>

you might want to get in touch with @martindurant then, he appears to be building a metadata / chunk location reader in [Maybe support hdf4 by martindurant · Pull Request #494 · fsspec/kerchunk · GitHub](https://github.com/fsspec/kerchunk/pull/494)

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [August 20, 2024, 1:21pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/28 "2024-08-20T13:21:00Z")

</div>

Yep, giving this a stab, and I can confirm that the internals of HDF4 are pure evil. That branch is not ready for anyone to use, more for those that want to actively help out. It does follow lined lists and find chunk tables, but doesn’t put things into groups (which array goes with which dimensions) yet and probably other stuff (how to INFLATE with zlib?).

---

<div class="post-metadata">

**Author:** ![rsignell](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rsignell/32/447_2.png) [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Post date:** [August 20, 2024, 2:59pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/29 "2024-08-20T14:59:47Z")

</div>

@jgallagher59701 is there HDF4 metadata extraction/transformation code in DMR++ that could be used here?

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [August 20, 2024, 3:16pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/30 "2024-08-20T15:16:09Z")

</div>

We have talked in the past about a dmr++ ↔ kerchunk bridge, but I don’t think there’s any work in that direction yet.

In principle, ~ v-zarr could ingest both ~ (this exists?) rather than converting to kerchunk, but that probably is about the same amount of work.

---

<div class="post-metadata">

**Author:** ![TomNicholas](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomnicholas/32/3189_2.png) [@TomNicholas](https://discourse.pangeo.io/u/TomNicholas)\
**Post date:** [August 20, 2024, 5:06pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/31 "2024-08-20T17:06:44Z")

</div>

> ~ v-zarr could ingest both ~ (this exists?)

Virtualizarr can ingest kerchunk references, or directly from files. See this comment

> <https://github.com/zarr-developers/VirtualiZarr/issues/218#issuecomment-2299337566>
>
> \*\*Let's list all the file formats that could potentially be represented efficien…tly as "virtual zarr" - i.e. zarr + chunk manifests.\*\*
> 
> The important criteria here is that the format must store data in a small number of contiguous chunks, such that access using http range requests to object storage is efficient. This rules out some formats, for example I don't think we can efficiently access this format that @kmuehlbauer mentioned over in https://github.com/openradar/xradar/issues/187#issuecomment-2271327041:
> 
> \> file formats where variables are written interleaved within one chunk of data (eg: 100 bytes v1, 100 bytes v2, 100 bytes v3, 100 bytes v1, 100 bytes v2, 100 bytes v3, ...)? Is there something like strides available?
> 
> If we start thinking of Zarr as a "SuperFormat" (super as in superset, not as in super-duper), then this is the list of existing formats comprising that set of what can be referenced using chunk manifests (see https://github.com/zarr-developers/zarr-specs/issues/287).
> 
> \---
> 
> \*\*Definitely can support:\*\*
> 
> \- \[x\] HDF5
> \- \[x\] NetCDF4
> \- \[x\] GRIB
> \- \[x\] TIFF
> \- \[x\] FITS
> \- \[x\] DMR++ https://github.com/zarr-developers/VirtualiZarr/issues/85
> \- \[\] Zarr itself! (a "Zarr of Zarrs")
> 
> \*\*Probably can support:\*\*
> 
> \- \[\] HDF4 https://github.com/zarr-developers/VirtualiZarr/issues/216
> \- \[\] arbitrary collections of \`.npz\` files
> 
> \*\*Maybe can support?\*\*
> 
> \- \[\] CSV https://github.com/zarr-developers/VirtualiZarr/issues/200
> \- \[\] Virtual Rasters https://github.com/zarr-developers/VirtualiZarr/issues/166
> \- \[\] rainbow https://github.com/openradar/xradar/issues/187#issuecomment-2271327041
> 
> \*\*Probably can't support:\*\*
> 
> \- furuno https://github.com/openradar/xradar/issues/187#issuecomment-2271327041
> \- sigmet/nexrad https://github.com/openradar/xradar/issues/187#issuecomment-2271327041
> 
> (The checkboxes indicate whether or not a working implementation already exists - going through kerchunks' in-memory format as an intermediate or creating a \`ManifestArray\` directly.)
> 
> cc @jhamman @d-v-b

Virtualizarr can also now ingest DMR++ (it just needs to be merged, probably upstream in `earthaccess` instead).

> <https://github.com/zarr-developers/VirtualiZarr/pull/113>
>
> \- \[x\] Closes #85
> \- \[x\] Tests added
> \- \[x\] Changes are documented in docs/releas…es.rst
> \- \[x\] New functions/methods are listed in api.rst
> \- \[x\] Open dataset via \`group=\` param
> \- \[x\] Use \[reader options\]( https://github.com/TomNicholas/VirtualiZarr/pull/67) for dmr file opening pattern

---

<div class="post-metadata">

**Author:** ![jgallagher59701](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jgallagher59701/32/2781_2.png) [@jgallagher59701](https://discourse.pangeo.io/u/jgallagher59701)\
**Post date:** [August 20, 2024, 6:14pm UTC](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690/32 "2024-08-20T18:14:05Z")

</div>

> [@rsignell](#):
>
> @jgallagher59701 is there HDF4 metadata extraction/transformation code in DMR++ that could be used here?

There is, on two levels. First, you can look at the DMR++ and the kinds of elements it uses to see various semantic details of HDF4 illustrated (e.g., a ‘chunk’ in hdf4 is not atomic, it can be made up of several discontiguous ‘blocks’ Yeah…).  
On another level, you can look at the C++ code we have to see how we work with these files.  
Bonus: To actually make most of NASA’s data usable, you need to use/understand the HDF4-EOS2 API since that is how the latitude and longitude values for the level 2 and 3 (swath and grid) data are computed.  
Disclaimer: I am not the author of this code, so for the real details will take me some time to dig up.

[Previous page](https://discourse.pangeo.io/t/making-kerchunk-as-simple-as-a-toggle/3690.md?page=1)