Where Do You Spend the Most Time in an ICESat-2 Workflow?

Science ICESat-2
1

Hi everyone,

I’ve been working with ICESat-2 datasets (particularly ATL20) for about a year now, and recently I’ve been thinking more broadly about the end-to-end workflow involved in working with these data, from discovery and access to preprocessing and analysis.

I’m curious to hear from others who regularly use ICESat-2 products:

  • Which parts of the workflow do you find most time-consuming?
  • What are the biggest challenges or frustrations you encounter?
  • Are there tasks that you find yourself repeatedly doing across different projects?

I’m interested in understanding how different researchers approach these datasets and where the major bottlenecks tend to arise.

Looking forward to hearing your thoughts. Thanks!

2

Hey Ram,

I’m not a regular ICESat-2 user, but did some work with Shane Grigsby on aggregations of point observations to DGGS using ATL-06 as a case study and can share a bit about the challenges that I ran into. I found that lock contention in h5coro led to poor performance when working with many files and that untuned-use led to memory bloating due to open file handles.

I think an async interface to HDF5 files could help a lot. I did a brief amount of experimentation in GitHub - virtual-zarr/async-hdf5: Experimenting with a Zarr proxy for HDF5, backed by Rust · GitHub but ran out of time to go farther. I’ve also been wondering for a while how fast and easy access could be if using virtual zarr and zarr-datafusion-search but similarly haven’t found time to experiment.

Cheers,
Max

1 Like
3

Hi Max,

Thanks a lot for the detailed response and for sharing those links - they’re incredibly helpful.

I wasn’t aware of the lock contention issues with h5coro, and the discussion around open file handles and memory bloat is particularly interesting. At the moment, I’m only working with relatively small ATL20 monthly products, so I haven’t run into these bottlenecks yet, but as I start expanding to the along-track ICESat-2 products (ATL03/06/08, etc.), I suspect they’ll become much more relevant.

Your point about an asynchronous interface to HDF5 is especially interesting. I’ve been thinking a lot about how to build tooling that scales beyond simple analyses, so I’ll definitely spend some time reading through the async-hdf5, Virtual Zarr, and zarr-datafusion-search projects that you linked.

Really appreciate you taking the time to share your experience - it gives me a much clearer idea of where some of the deeper infrastructure challenges lie.

Thanks again!

1 Like