# Cloud computing using NASA Earthdata with Earthdata login

**URL:** <https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434>\
**Category:** Cloud\
**Created:** [April 28, 2022, 1:59pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434 "2022-04-28T13:59:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jinbow](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jinbow/32/1242_2.png) [@jinbow](https://discourse.pangeo.io/u/jinbow)\
**Post date:** [April 28, 2022, 1:59pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/1 "2022-04-28T13:59:17Z")

</div>

Hey All,

Much of NASA Earthdata especially those from PO.PAAC are now hosted on S3. Using parallel computing is the key to unlock the potential of hosting the data in the cloud. I could not find much information here, but has anybody used Dask (or MPI) using an AWS cluster directly on NASA Earthdata which requires a (free) Earthdata login (EDL)? How was your experience?

Jinbo

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [April 28, 2022, 4:26pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/2 "2022-04-28T16:26:34Z")

</div>

@scottyhq has an example at [Skip the download! Stream NASA data directly into Python objects | by Scott Henderson | pangeo | Medium](https://medium.com/pangeo/intake-stac-nasa-4cd78d6246b7#1528). I’m not sure if that’s the state of the art or not, but it might get you started.

---

<div class="post-metadata">

**Author:** ![jinbow](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jinbow/32/1242_2.png) [@jinbow](https://discourse.pangeo.io/u/jinbow)\
**Post date:** [April 28, 2022, 7:35pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/3 "2022-04-28T19:35:42Z")

</div>

Thx @scottyhq, it is useful, but it did not touch Dask or parallel computing. Not sure whether it is too trivial.

---

<div class="post-metadata">

**Author:** ![scottyhq](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/scottyhq/32/162_2.png) [@scottyhq](https://discourse.pangeo.io/u/scottyhq)\
**Post date:** [April 28, 2022, 8:23pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/4 "2022-04-28T20:23:27Z")

</div>

Hi, yes, that example is now a bit dated and doesn’t go into dask. You might also have a look at materials from the more recent ICESat-2 hackweek: [Cloud Computing Tutorial — ICESat-2 Hackweek 2022](https://icesat-2.hackweek.io/tutorials/cloud-computing/cloud-computing-tutorial.html#exercise-3-access-cog-data-from-s3-using-earthdata-search)

Not sure what dataset you’re interested in. Working with dask on a single machine is straightforward because you can read credentials from a local file (~/.netrc), but on a distributed cluster you need some way of moving credentials (ideally temporary ones) across machines. Here is another example for raster data using data from LPDAAC [CloudDAAC\_Binders/s3\_v\_http.ipynb at main · rmg55/CloudDAAC\_Binders · GitHub](https://github.com/rmg55/CloudDAAC_Binders/blob/main/s3_v_http.ipynb)

---

<div class="post-metadata">

**Author:** ![jinbow](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jinbow/32/1242_2.png) [@jinbow](https://discourse.pangeo.io/u/jinbow)\
**Post date:** [April 29, 2022, 6:08am UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/5 "2022-04-29T06:08:27Z")

</div>

I took a look at the tutorial. It is very useful for laying out the process. But it seemed to be more complicated than necessary. Is this the reason why not many existing tutorials use earthdata to demo large-scale computation? What is your experience @rabernat?

@scottyhq I am interested in using cluster with earthdata in general. MUR SST at 1km is one example for its size.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [April 29, 2022, 1:19pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/6 "2022-04-29T13:19:25Z")

</div>

Fortunately a completely public version of MUR exists outside the Earthdata enclave: [Multi-Scale Ultra High Resolution (MUR) Sea Surface Temperature (SST) - Registry of Open Data on AWS](https://registry.opendata.aws/mur/).

I don’t have any insights on Earthdata login that Scott and Tom do not. We have usually been able to make it work using fsspec to pass through the credentials

It would be great if someone would write a definitive guide on how to use Earthdata data with Pangeo. Why not just as a forum post here on Discourse?

> [@jinbow](#):
>
> But it seemed to be more complicated than necessary.

To be honest, this is an issue to raise with NASA, not Pangeo. NASA has created a pretty complicated wall around their data with Earthdata login. We are doing our best to deal with it given those constraints.

---

<div class="post-metadata">

**Author:** ![JessicaS11](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jessicas11/32/229_2.png) [@JessicaS11](https://discourse.pangeo.io/u/JessicaS11)\
**Post date:** [May 20, 2022, 9:01pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/7 "2022-05-20T21:01:42Z")

</div>

> [@rabernat](#):
>
> It would be great if someone would write a definitive guide on how to use Earthdata data with Pangeo.

A belated reply that this is on my and @betolink’s long to-do list, because it’s a pain point for icepyx as well (I know Luis’ [earthdata library](https://github.com/nsidc/earthdata) handles S3 authentication, but not sure if its set up for parallel computing). I’m anticipating that I can dig in on this sometime this summer (we have a very crude start [here](https://github.com/icesat2py/icepyx/tree/login_docs)).

---

<div class="post-metadata">

**Author:** ![jhkennedy](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhkennedy/32/709_2.png) [@jhkennedy](https://discourse.pangeo.io/u/jhkennedy)\
**Post date:** [May 20, 2022, 10:00pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/8 "2022-05-20T22:00:01Z")

</div>

@JessicaS11 and @betolink , I’ve also had this on my todo list – I’d be happy to team up this summer if you want to organized a mini-hackathon of sorts. 😄

---

<div class="post-metadata">

**Author:** ![sharkinsspatial](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/sharkinsspatial/32/992_2.png) [@sharkinsspatial](https://discourse.pangeo.io/u/sharkinsspatial)\
**Post date:** [May 23, 2022, 5:17pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/9 "2022-05-23T17:17:50Z")

</div>

@JessicaS11 @jhkennedy I’d also be interested in assisting with developing a guide for this if we have plans for tackling this later this summer. As a side note, there are several ongoing internal NASA efforts to improve the access mechanics of authentication for DAAC managed object storage. Some of the considerations are covered in this recent presentation [Token Expiration Whitepaper - Google Docs](https://docs.google.com/document/d/18GyoMZj0I2HKAXwqyeziO0ISbOwHxo1TN4eAlR4mH3U/view)

---

<div class="post-metadata">

**Author:** ![jhkennedy](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhkennedy/32/709_2.png) [@jhkennedy](https://discourse.pangeo.io/u/jhkennedy)\
**Post date:** [May 24, 2022, 3:20pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/10 "2022-05-24T15:20:26Z")

</div>

@sharkinsspatial I’m at ASF DAAC and work on HyP3 so know a bit about those efforts, and I can definitely say I personally endorse that whitepaper’s conclusions. I’d be happy to know if there’s any support I can lend to it.

---

<div class="post-metadata">

**Author:** ![jinbow](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jinbow/32/1242_2.png) [@jinbow](https://discourse.pangeo.io/u/jinbow)\
**Post date:** [June 3, 2022, 4:39pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/11 "2022-06-03T16:39:02Z")

</div>

We here at JPL/PODAAC have been exploring this. So far we have made progress using AWS lambda to parallel process granule-based calculation, but no luck on Dask. I knew several other groups are exploring this too. It will be a fun summer project.

---

<div class="post-metadata">

**Author:** ![jinbow](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jinbow/32/1242_2.png) [@jinbow](https://discourse.pangeo.io/u/jinbow)\
**Post date:** [June 14, 2022, 7:15pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/12 "2022-06-14T19:15:44Z")

</div>

I made a baby step on this topic. Here is a notebook. The next target is using Dask to lazy-load POCLOUD MEaSUREs-SSH level-4 gridded product and perform distributed calculation.

> <https://github.com/podaac/the-coding-club/blob/main/notebooks/MEaSUREs-SSH-dask.ipynb>

---

<div class="post-metadata">

**Author:** ![betolink](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/betolink/32/247_2.png) [@betolink](https://discourse.pangeo.io/u/betolink)\
**Post date:** [June 16, 2022, 8:57pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/13 "2022-06-16T20:57:49Z")

</div>

Same here, this has been on my list for quiet some time now. One of the things to distinguish here is the use of a local cluster vs distributed cluster. For a local cluster case, I think what Scott did should work out of the box, for a distributed cluster… things could get a bit tricky with the S3 keys to the worker nodes and the fact that they expire after 1 hour etc. It would be cool if NASA could host a hack week focused only on access patterns to EDL-protected data both direct S3 access and on-prem from the DAACs. As Ryan mentioned, this is more of a NASA problem than a Pangeo issue. Nonetheless, It would be awesome to have some notebooks and/or a blog post on how to actually scale NASA dataset processing with Dask. I’m happy to team up with all of you on this.

---

<div class="post-metadata">

**Author:** ![jhkennedy](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhkennedy/32/709_2.png) [@jhkennedy](https://discourse.pangeo.io/u/jhkennedy)\
**Post date:** [June 17, 2022, 7:06pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/14 "2022-06-17T19:06:32Z")

</div>

> We have usually been able to make it work using fsspec to pass through the credentials

@rabernat `fsspec` does seem to have all the stuff needed. I’ve been kicking around making an `edlfs` package al la `s3fs` with @betolink. Idea would be to handle dealing with the credentials for uses (over HTTPS as that’ll should work with all DAAC systems). Does that seem like a reasonable idea? Also makes me wonder if we could get that work funded some how…

Really, EDL has prompted a ton of engineering work to fundamentally solve a NASA policy problem, so I’m hoping internal pushes to change NASA policy would make that package obsolete… but I have no over/under on how likely a policy change would be (well above my pay grade).

---

<div class="post-metadata">

**Author:** ![sharkinsspatial](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/sharkinsspatial/32/992_2.png) [@sharkinsspatial](https://discourse.pangeo.io/u/sharkinsspatial)\
**Post date:** [August 4, 2022, 7:47pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/15 "2022-08-04T19:47:50Z")

</div>

@jhkennedy Apologies I had lost track of this thread but just stumbled on [edlfs](https://github.com/NASA-Openscapes/edlfs) repo that you and @betolink were working on randomly while searching for something else in Openscapes :]. I’m out on vacation this week at the moment and travelling at the beginning of September. What do folks think about trying to organize a sprint around this in mid September?

If you’d like to pursue the `fs` implementation approach, we’d likely need to find some contributors with deep `aibotocore` experience. I’m a bit out of my depth determining how to implement timer based token refresh when working with the context manager pattern, but I’m sure there are folks lurking here who would have good recommendations around this.

---

<div class="post-metadata">

**Author:** ![betolink](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/betolink/32/247_2.png) [@betolink](https://discourse.pangeo.io/u/betolink)\
**Post date:** [August 4, 2022, 8:41pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/16 "2022-08-04T20:41:52Z")

</div>

I like the mid September idea. After some initial prototyping I don’t know if a full-flagged fsspec backend implementation is required to circumvent the EDL issue but it could be an opportunity for improvements.  
Joe suggested another workflow(involving [TEA](https://raw.githubusercontent.com/asfadmin/thin-egress-app/master/docs/images/tea.png)) that might just work to avoid the per-DAAC credentials and agree that if this goes forward as a package we’ll need aiobotocore/aiohttp expertise.

cc @JessicaS11 @jhkennedy

---

<div class="post-metadata">

**Author:** ![jhamman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhamman/32/34_2.png) [@jhamman](https://discourse.pangeo.io/u/jhamman)\
**Post date:** [August 5, 2022, 5:08pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/17 "2022-08-05T17:08:21Z")

</div>

> I like the mid September idea. After some initial prototyping I don’t know if a full-flagged fsspec backend implementation is required to circumvent the EDL issue but it could be an opportunity for improvements.

@martindurant, maybe you can help advise on whether a new file system is the right path here?

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [August 6, 2022, 12:27am UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/18 "2022-08-06T00:27:42Z")

</div>

There has been some talk at s3fs about making credentials which can renew themselves after becoming invalid. But I don’t know anything about the specifics of EDL (sorry, haven’t read the thread in detail yet).

Until then, I imagine, for something that is s3-except-auth, you might be best of subclassing S3FileSystem, which already has almost everything you need. That should be much easier than writing a new backend from scratch, although that’s not _too_ hard either.

---

<div class="post-metadata">

**Author:** ![jhkennedy](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhkennedy/32/709_2.png) [@jhkennedy](https://discourse.pangeo.io/u/jhkennedy)\
**Post date:** [August 6, 2022, 3:15pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/19 "2022-08-06T15:15:52Z")

</div>

> What do folks think about trying to organize a sprint around this in mid September?

I think that’s a great idea! There’s at least 3 of us at ASF that’d be interested in participating, including @forrestfwilliams .

There’s two access patterns for NASA DAACs in this space:

1. Using EDL and HTTPS only. Stuff is effectively distributed via a DAAC URL that’s redirected to:  
a. A CloudFront URL or an S3 signed HTTPS URL if you’re in-region for cloud-hosted data  
b. An on-prem URL

2. By hitting a DAAC specific endpoint, you can get temporary AWS access keys that give you read-only access

I think targeting 1 might make the most sense initially as that’ll mostly just be aiohttp based. Getting 2 working will definitely need self-renewing credentials; maybe it’d make sense to focus some dev. time on adding that to s3fs before adding in the EDL complications? Though a general s3fs solution might be much bigger.

---

<div class="post-metadata">

**Author:** ![martindurant](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/martindurant/32/388_2.png) [@martindurant](https://discourse.pangeo.io/u/martindurant)\
**Post date:** [August 8, 2022, 2:02pm UTC](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434/20 "2022-08-08T14:02:01Z")

</div>

This notebook shows earthdata login with s3fs and kerchunk: [https://nbviewer.org/github/cgentemann/cloud\_science/blob/master/zarr\_meta/cloud\_mur\_v41\_benchmark.ipynb](https://nbviewer.org/github/cgentemann/cloud_science/blob/master/zarr_meta/cloud_mur_v41_benchmark.ipynb) , but I assume all operations here were fact enough not to worry about expiry.

[Next page](https://discourse.pangeo.io/t/cloud-computing-using-nasa-earthdata-with-earthdata-login/2434.md?page=2)