# Seeking advice on file format for an API

**URL:** <https://discourse.pangeo.io/t/seeking-advice-on-file-format-for-an-api/4883>\
**Category:** Cloud\
**Created:** [February 10, 2025, 3:53pm UTC](https://discourse.pangeo.io/t/seeking-advice-on-file-format-for-an-api/4883 "2025-02-10T15:53:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ags\_pangeo](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@ags\_pangeo](https://discourse.pangeo.io/u/ags_pangeo)\
**Post date:** [February 10, 2025, 3:53pm UTC](https://discourse.pangeo.io/t/seeking-advice-on-file-format-for-an-api/4883/1 "2025-02-10T15:53:42Z")

</div>

I’m building out an internal API that will serve temperature data at various altitudes on a predetermined grid (x deg lat x y deg long x z km alt). I have data at cadence of n minutes for this grid.

I’ve worked with this kind of data - what would be an ideal format to store such data, such that I can use metadata to pull the exact partition that the latitude, longitude, altitude, and timestamp can be found?

I considered GRIB2 files - partitioned by timestamp and altitude, resulting in small files with temp\_timestamp\_altitude.grib as names… the API backend that is able to quickly query AWS s3 for the right file. However, this results in 1440/n timestamps \* y altitudes files, which is quite clunky.

Would someone here have recommendations for a cloud optimized format that can be used to back a sub-500ms response time API?

NOTE: eventually, we’ll extend the API to accept lat/long/alt values that do not sit exactly on the grid points, so we’ll find the closest and then interpolate values.

Maybe postgis?

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [February 10, 2025, 5:19pm UTC](https://discourse.pangeo.io/t/seeking-advice-on-file-format-for-an-api/4883/2 "2025-02-10T17:19:23Z")

</div>

Welcome @ags_pangeo!

I think the formula that our community has converged around for optimal performance and flexibility is Zarr. Combined with XPublish’s EDR plugin, this gives you a great architecture to achieve your needs. In fact, @jhamman and Alex Kerney recently gave a presentation on this exact subject at the Pangeo showcase.

> [@Pangeo Showcase: "Xpublish at Scale(s)"](https://discourse.pangeo.io/t/pangeo-showcase-xpublish-at-scale-s/4825):
>
> [[Xpublish at Scale(s)] ](https://www.youtube.com/watch?v=6VnBSUTmIKY) Title: “Xpublish at Scale(s)” Invited speakers: Alex Kerney (ORCID: [0000-0002-0484-9306](https://orcid.org/0000-0002-0484-9306)) and Joe Hamman (ORCID: [0000-0001-7479-8439](https://orcid.org/0000-0001-7479-8439)) When: Wednesday, January 29, 2025 at 4 PM EST Where: [Launch Meeting - Zoom](https://numfocus-org.zoom.us/j/83234473026?pwd=mgM3uAuru1AE1aFPlRjW2dAAAyo8IY.1) Abstract: A (re-) introduction to Xpublish and the ability to scale serving Xarray data from community to commercial. Agenda: ~15 minutes - Showcase presentation 10 - 30 minutes - Discussion 15 - 30 minutes - Community check-in

If you’re interested in a hosted solution, rather than something you have to build and manage yourself, our company Earthmover offers a subscription-based managed service using this type of architecture.

> **[EDR: Environmental Data Retrieval | Earthmover](https://docs.earthmover.io/query-service/edr)**
>
> EDR is currently in private beta. Features and capabilities may change with time.

---

<div class="post-metadata">

**Author:** ![ags\_pangeo](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@ags\_pangeo](https://discourse.pangeo.io/u/ags_pangeo)\
**Post date:** [February 10, 2025, 5:20pm UTC](https://discourse.pangeo.io/t/seeking-advice-on-file-format-for-an-api/4883/3 "2025-02-10T17:20:20Z")

</div>

That’s pretty neat! I’ll take a look at all of the links (including the hosted solution) today.