# Pangeo Showcase: "HDF5 at the Speed of Zarr"

**URL:** <https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084>\
**Category:** Pangeo Showcase\
**Created:** [March 6, 2024, 11:31am UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084 "2024-03-06T11:31:01Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![rsignell](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rsignell/32/447_2.png) [@rsignell](https://discourse.pangeo.io/u/rsignell)\
**Post date:** [March 6, 2024, 11:31am UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/1 "2024-03-06T11:31:01Z")

</div>

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10830528.svg)](https://doi.org/10.5281/zenodo.10830528)

[![](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/8/8a9ba37f7df79f9c27acd3efc3f10aae75af597a.jpeg "HDF5 at the speed of Zarr") ](https://www.youtube.com/watch?v=iRboOFIB74o)

**Title:** “HDF5 at the Speed of Zarr”  
**Invited Speaker:** Luis Lopez (ORCID:[0000-0003-4896-3263](https://orcid.org/0000-0003-4896-3263))  
**When:** Wednesday March 13, 4PM EDT  
**Where:** [Launch Meeting - Zoom](https://numfocus-org.zoom.us/j/83234473026?pwd=mgM3uAuru1AE1aFPlRjW2dAAAyo8IY.1)  
**Abstract:** As flexible and powerful as HDF5 can be, it comes with big tradeoffs when it’s accessed from remote storage systems, mainly because the file format and the client I/O libraries were designed for local and supercomputing workflows. As scientific data and workflows migrate to the cloud, efficient access to data stored in HDF5 format is a key factor that will accelerate or slow down “science in the cloud” across all disciplines.

We have been working on testing implementation of recently available features in the HDF5 stack that results in performant access to HDF5 from remote cloud storage. This performance seems on par with modern cloud-native formats like Zarr but with the advantage of not having to reformat the data or generate metadata sidecar files (DMR++, Kerchunk).

- 20 minutes - Community Showcase
- 40 minutes - Showcase Discussion/Community Check-ins

---

<div class="post-metadata">

**Author:** ![betolink](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/betolink/32/247_2.png) [@betolink](https://discourse.pangeo.io/u/betolink)\
**Post date:** [March 13, 2024, 10:38pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/2 "2024-03-13T22:38:03Z")

</div>

Here is a list of useful links for this presentation:

[Slides](https://docs.google.com/presentation/d/1iYFvGt9Zz0iaTj0STIMbboRKcBGhpOH_LuLBLqsJAlk/edit?usp=sharing)

[H5Cloud project repository](https://github.com/ICESAT-2HackWeek/h5cloud)  
[IS2 cloud access](https://gist.github.com/betolink/b545c364f80882c113b8cc27b763c729)  
[Discussion on HDF performance](https://github.com/nsidc/earthaccess/discussions/251)

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [March 15, 2024, 3:41pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/3 "2024-03-15T15:41:09Z")

</div>

Just watching your talk, sorry I missed it. Thanks for doing this!

“Access patterns matter” - couldn’t agree more.

All HDF5 files ought to be written in “consolidated” form with the metadata up front! 🙂

Some thoughts:

- does your “cloud optimised” also include rechunking the data for some preferred use case?
- “first” is always the recommended cache pattern for HDF5 in any of the cloud FS backends. This is what kerchunk, for instance, uses all the time. There has been an argument that fsspec should try to choose the right cache per file type, but “readahead” is still the best for stream rather than random access. Also, the cache size could be bigger, but that is also usage dependent.
- I wonder if you tried tuning anything for benchmarking with kerchunk (of course I have extra knowledge here)
- in your position, I would make some explicit comparison of the pros/cons of rewriting the data in C.O.HDF5 versus rewriting in zarr

---

<div class="post-metadata">

**Author:** ![betolink](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/betolink/32/247_2.png) [@betolink](https://discourse.pangeo.io/u/betolink)\
**Post date:** [March 17, 2024, 1:13am UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/4 "2024-03-17T01:13:51Z")

</div>

Hi Martin, thanks for watching!

> does your “cloud optimised” also include rechunking the data for some preferred use case?

For the tests we ran, we did not alter the original chunking of the data. It’s definitely a factor when we optimize access. It’s a tricky question right, too small and we get a lot of I/O, too big and we run into other issues like memory and decompression speeds.

> “first” is always the recommended cache pattern for HDF5 in any of the cloud FS backends.

I think this is really important, I wonder if packages like earthaccess, pystac-client, xarray, etc can take the _recommended_ and transform it into _enforced_. Almost nobody knows why this is important for cloud access.

> Kerchunk

Not formally, I noticed that Kerchunk was way faster with cloud-optimized HDF5 but for that I needed to tell it how to open the file (cache first, size of the buffer = size of the metadata) I forked kerchunk and modified it a bit to allow this.

> Re-writing in Zarr vs CO-HDF5

I think, (personal opinion) that if our data model fits in Zarr, we should use it! However, in the context of remote sensing data, there are plenty of mission requirements that make it hard to depart from HDF. In these cases re-writing CO-HDF5 could be a better bang for the buck. I think that comparison needs to be written, perhaps in the form of a short technical paper…

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [March 18, 2024, 3:00pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/5 "2024-03-18T15:00:09Z")

</div>

> [@betolink](#):
>
> [chunking is] a tricky question right

I would say there is no “right”, only optimal cases for each access pattern. There are, however, some very wrong cases 🙂

> earthaccess, pystac-client, xarray, etc can take the _recommended_ and transform it into _enforced_ .

I don’t see why they shouldn’t have decent defaults, allowing users to override them if necessary.

> I noticed that Kerchunk was way faster with cloud-optimized HDF5

I meant for final reading of the dataset, not the scanning phase - but this can be important too

> I forked kerchunk and modified it a bit to allow this.

Feel free to share/propose any PR. I wonder, is there a way to know the size of the metadata area, i.e., the bit it’s worth caching, before starting to scan a file?

---

<div class="post-metadata">

**Author:** ![chuckwondo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chuckwondo/32/2170_2.png) [@chuckwondo](https://discourse.pangeo.io/u/chuckwondo)\
**Post date:** [June 6, 2024, 10:29pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/6 "2024-06-06T22:29:07Z")

</div>

> “first” is always the recommended cache pattern for HDF5 in any of the cloud FS backends.

Is this under the assumption that the metadata is “consolidated” up front? If not, it seems odd to me that this would be the recommended cache type for files without consolidated metadata up front.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [June 7, 2024, 1:22am UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/7 "2024-06-07T01:22:27Z")

</div>

> Is this under the assumption that the metadata is “consolidated” up front?

No; I have found that even without explicit consolidation, the bulk of the metadata tends to be in the first few MB, and the reader will seek there repeatedly. The rest of the metadata may be scattered anywhere amongst the data chunks, so there is no reasonable way to readahead or otherwise cache/pre-fetch for those.

---

<div class="post-metadata">

**Author:** ![chuckwondo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chuckwondo/32/2170_2.png) [@chuckwondo](https://discourse.pangeo.io/u/chuckwondo)\
**Post date:** [June 7, 2024, 5:09pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/8 "2024-06-07T17:09:04Z")

</div>

> No; I have found that even without explicit consolidation, the bulk of the metadata tends to be in the first few MB, and the reader will seek there repeatedly. The rest of the metadata may be scattered anywhere amongst the data chunks, so there is no reasonable way to readahead or otherwise cache/pre-fetch for those.

In my experiments with GEDI files, where I’m performing subsetting, I’m finding several other cache types to be _much_ better than the “first” cache type. Specifically, I’m finding “all,” “blockcache,” “background,” and “mmap” to all be roughly 30% more performant than “first.”

I’m running more comprehensive tests using various combinations of cache type and block size to gather some performance statistics, so I’m happy to share once I have more numbers.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [June 7, 2024, 6:06pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/9 "2024-06-07T18:06:39Z")

</div>

OK, look forward to seeing your results.

---

<div class="post-metadata">

**Author:** ![betolink](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/betolink/32/247_2.png) [@betolink](https://discourse.pangeo.io/u/betolink)\
**Post date:** [June 10, 2024, 3:29pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/10 "2024-06-10T15:29:58Z")

</div>

> I’m running more comprehensive tests using various combinations of cache type and block size to gather some performance statistics, so I’m happy to share once I have more numbers.

This is great news! I was thinking of doing the same with random samples from the NASA DAACs, I think the cache configuration could be even “smarter” if we know the chunking in advance (via kerchunk or dmrpp). cc @TomNicholas @ayushnag

---

<div class="post-metadata">

**Author:** ![chuckwondo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chuckwondo/32/2170_2.png) [@chuckwondo](https://discourse.pangeo.io/u/chuckwondo)\
**Post date:** [June 24, 2024, 6:43pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/11 "2024-06-24T18:43:05Z")

</div>

Without getting into much detail, the following chart shows how the various combinations of `cache_type`, `block_size` (in MiB), and `fill` (bool) arguments to `s3fs.S3FileSystem.open` affect read performance of `.h5` files.

I gathered these metrics in the context of running an algorithm for subsetting a set of `.h5` GEDI L4A files in basic Python multiprocessing code (i.e., no use of any specific libraries for scaling, such as dask, ray, etc.). The gist of the code is that it takes a list of `.h5` files in S3 (in this case, ~1200), and for each file it reads (via `h5py`) a handful of datasets (the same for each file) out of the dozens available, doing so across 32 CPUs, each with 64GiB of RAM (far more than actually needed).

We also compare this to simply downloading the files and reading them from the local filesystem, rather than reading directly from S3. This case is where the y-axis tick is labeled `('download', 0.0, False)`. I ran each combination ~30 times each (some combos had 1 or 2 jobs fail).

As you can see, all uses of cache type `first` perform significantly worse than anything else, including downloading. Of course, the `.h5` files are _not_ cloud-optimized, so nothing performs more than marginally better than downloading, but that’s sort of the point.

I’m no expert on any of this stuff, but I’m happy to provide more details if anybody has questions.

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/0/06123b71b90e3fdc676809174138ea8fa422698d.png)

---

<div class="post-metadata">

**Author:** ![betolink](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/betolink/32/247_2.png) [@betolink](https://discourse.pangeo.io/u/betolink)\
**Post date:** [June 26, 2024, 3:01pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/12 "2024-06-26T15:01:16Z")

</div>

Thanks for this @chuckwondo! I think having the default cache (read-ahead) would be great to complete this chart. Is this on Github?

---

<div class="post-metadata">

**Author:** ![chuckwondo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/chuckwondo/32/2170_2.png) [@chuckwondo](https://discourse.pangeo.io/u/chuckwondo)\
**Post date:** [June 26, 2024, 3:57pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/13 "2024-06-26T15:57:25Z")

</div>

Sorry @betolink, I forgot to mention that I didn’t include the other cache types because my initial runs had already shown them to perform poorly. The ones I’ve included here, other than “first,” are the ones that showed clearly better performance than the other types, including “first.” I ended up including “first” _only_ because of this discussion, so I could show that it should _not_ necessarily be the recommended cache pattern for HDF5.

However, if you’d like to see proof of the poor performance (in my use case, of course) for “readahead” and some others, I’d be happy to run more jobs to gather stats. Just let me know.

The performance data is not on GitHub, as these tests were run (and the metrics were captured) on a non-public system. However, I should be able to pull together a MRE that should be runnable outside of this system, requiring only Earthdata Login credentials, which is actually an issue for enhancement that we have, so others can run the GEDI Subsetting code outside of our current system.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [August 13, 2024, 1:22pm UTC](https://discourse.pangeo.io/t/pangeo-showcase-hdf5-at-the-speed-of-zarr/4084/14 "2024-08-13T13:22:44Z")

</div>

While scanning files, we generally were recommending “first” caching, as the compromise between time and total bytes downloaded. But it depends on the files being scanned!