# Cleaning out the pangeo-data google cloud storage bucket

**URL:** <https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353>\
**Category:** Cloud\
**Created:** [November 12, 2019, 3:52am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353 "2019-11-12T03:52:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 12, 2019, 3:52am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/1 "2019-11-12T03:52:51Z")

</div>

When Pangeo got its first funding from the NSF Earthcube project (starting in October 2017), we received $100K of credits on Google Cloud Platform via the NSF BIGDATA program. We knew almost nothing about cloud, and needless to say, we have learned a lot.

When we were first starting out, we took a liberal approach to costs. Our goal was to experiment and learn. We gave dozens of people write access to our cloud storage, and we accumulated hundreds of different datasets. Almost all of these went into a single Google Cloud Storage bucket: `pangeo-data`.

That NSF project, and the cloud credits that came with it, are coming to an end. We need to take stock of the cloud data we have accumulated and make a plan for transitioning to a more sustainable future. This will involve moving and / or deleting much of the data we currently own. In particularly, we plan to delete the `pangeo-data` bucket and transition to a policy of “one bucket per data provider.”

An important step is to catalog the data in our intake data catalog. We would like all our data to eventually end up in catalog, rather than just living in a random, un-discoverable location in cloud storage. Our current catalog is here:  
[https://pangeo-data.github.io/pangeo-datastore/](https://pangeo-data.github.io/pangeo-datastore/)

@jhamman has compiled a list of datasets that are in our bucket but not in our catalog. We have put this information into a google spreadsheet:

> **[Pangeo GCS Data Cleanup](https://docs.google.com/spreadsheets/d/1O-4LIPe6vZ2d0qiVQru3Xdk9hNbwnYyTW9gZW6mOJYo/edit#gid%3D0)**
>
> Sheet1
> 
> bucket,p1,Zarr Datasets,Who?,Info
> pangeo-data,CMIP5-ts,132,Pending Removal
> pangeo-data,CMIP6,1,Move to new bucket. Needs cataloging.
> pangeo-data,CMIP6-test,3,Keep in place. Needs...

Our goal over the next week is to categorize all these datasets into one of three categories:

- Remove
- Move to new bucket (not `pangeo-data`) and add to catalog
- Keep in existing bucket and add to catalog

Below is a summary of the un-accounted-for datasets. If you recognize any of these, please open up the google sheet and claim it / help decide its fate.

**All unclaimed datasets will be deleted on Friday, Nov. 22.**

When deciding whether a dataset should be kept or deleted, we should consider several factors:

- How big / expensive is it?
- Is it being actively used?
- Can the data easily be regenerated from another public archive?

### Summary of Unclaimed Datasets

| pangeo-data | CMIP5-ts | # datasets |
| --- | --- | --- |
| pangeo-data | CMIP6 | 1 |
| pangeo-data | CMIP6-test | 3 |
| pangeo-data | ECCO.zarr | 1 |
| pangeo-data | ECCO\_chank.zarr | 1 |
| pangeo-data | ECCO\_layers.zarr | 1 |
| pangeo-data | GEOS\_V2p1 | 18 |
| pangeo-data | GEOS\_V2p1\_L | 18 |
| pangeo-data | NATL60-CJM165-SSH-1h-1m2deg2deg | 1 |
| pangeo-data | NATL60-CJM165-SSH-1h-2D | 1 |
| pangeo-data | NATL60-CJM165-SSU-1h-1m2deg2deg | 1 |
| pangeo-data | NATL60-CJM165-SSV-1h-1m2deg2deg | 1 |
| pangeo-data | SLaPS | 216 |
| pangeo-data | amazonas | 1 |
| pangeo-data | avhrr-patmos-x-cloudprops-noaa-asc-fc\_TESTING4 | 1 |
| pangeo-data | balwada | 4 |
| pangeo-data | cesm | 4 |
| pangeo-data | channel | 1 |
| pangeo-data | cm2.6 | 3 |
| pangeo-data | dataset-duacs-rep-global-merged-allsat-phy-l4-v3 | 1 |
| pangeo-data | eNATL60-BLB002-SSU-1h | 1 |
| pangeo-data | eNATL60-BLB002-SSV-1h | 1 |
| pangeo-data | eNATL60-BLBT02-SSU-1h | 1 |
| pangeo-data | eNATL60-BLBT02-SSV-1h | 1 |
| pangeo-data | eNATL60-BLBT02X-ssh | 2 |
| pangeo-data | eNATL60-I | 3 |
| pangeo-data | eORCA025-I | 1 |
| pangeo-data | eORCA1-I | 1 |
| pangeo-data | ecco | 11 |
| pangeo-data | esgf\_test | 1 |
| pangeo-data | gpm\_imerg | 4 |
| pangeo-data | gross | 1372 |
| pangeo-data | kai-llc4320-vertical-fluxes | 7 |
| pangeo-data | llc4320 | 1 |
| pangeo-data | model\_vars\_5day\_av\_zarr | 1 |
| pangeo-data | netcdf\_test\_data | 1 |
| pangeo-data | polar | 4 |
| pangeo-data | pyqg | 10 |
| pangeo-data | rsignell | 17 |
| pangeo-data | storage-benchmarks | 3 |
| pangeo-data | test | 1 |
| pangeo-data | tracer\_vars\_5day\_av\_zarr | 1 |
| pangeo-data | tracmip | 7067 |
| pangeo-data | xESMF\_test | 3 |
| pangeo-data | zarr-eNATL60 | 6 |
| pangeo-data | zarr\_NATL60-CJM165\_SSU\_1h\_y2013m07-09 | 1 |
| pangeo-ecco | llc | 3 |
| pangeo-ocean-ml | LLC4320 | 124 |
| pangeo-parcels | med\_sea\_connectivity\_v2019.09.11.2 | 2 |

---

<div class="post-metadata">

**Author:** ![jhamman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhamman/32/34_2.png) [@jhamman](https://discourse.pangeo.io/u/jhamman)\
**Post date:** [November 12, 2019, 3:26pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/2 "2019-11-12T15:26:10Z")

</div>

Pinging a bunch of people I want to make sure see this: @scottyhq, @rsignell, @naomi-henderson, @jbusecke, @jlesommer, @dhruvbalwada, @davidbrochart.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 12, 2019, 3:39pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/3 "2019-11-12T15:39:06Z")

</div>

Perhaps we could also generate an email list of people with storage-admin rights?

---

<div class="post-metadata">

**Author:** ![jbusecke](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jbusecke/32/118_2.png) [@jbusecke](https://discourse.pangeo.io/u/jbusecke)\
**Post date:** [November 12, 2019, 5:23pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/4 "2019-11-12T17:23:24Z")

</div>

The ‚CMIP6‘ data is not the one we worked from at the hackathon I assume? Unless that is the case I have nothing to claim at this point. Thanks for the ping though.

---

<div class="post-metadata">

**Author:** ![willirath](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/willirath/32/122_2.png) [@willirath](https://discourse.pangeo.io/u/willirath)\
**Post date:** [November 20, 2019, 7:42am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/5 "2019-11-20T07:42:44Z")

</div>

If possible, I’d like to keep the approx. 300 GB (?) `pangeo-parcels` bucket around. The data are 40 days (~1000 hourly snapshots) of 15.000.000 Lagrangian particles in the Med Sea and I’m working on ways to generate fast visualisations from these trajectories.

---

<div class="post-metadata">

**Author:** ![jhamman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jhamman/32/34_2.png) [@jhamman](https://discourse.pangeo.io/u/jhamman)\
**Post date:** [November 20, 2019, 8:19am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/6 "2019-11-20T08:19:30Z")

</div>

Sounds good @willirath. Can you add that data to the pangeo-datastore catalog?

---

<div class="post-metadata">

**Author:** ![Aurelie\_ALBERT](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/aurelie_albert/32/210_2.png) [@Aurelie\_ALBERT](https://discourse.pangeo.io/u/Aurelie_ALBERT)\
**Post date:** [November 28, 2019, 4:18pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/7 "2019-11-28T16:18:03Z")

</div>

Hi, I am currently documenting and reorganizing the data I (on behalf of MEOM team in Grenoble) put on the cloud in order to fill the pangeo-datastore catalog (sorry it took so long to do it) and I was wondering if a dedicated bucket pangeo-nemo should be created ? Thanks

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [November 29, 2019, 2:10pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/8 "2019-11-29T14:10:58Z")

</div>

Yes, it would be very convenient to have a dedicated bucket for each group / project. I will create one for you.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [December 2, 2019, 6:07pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/9 "2019-12-02T18:07:57Z")

</div>

I have created the `pangeo-meom` bucket for you @Aurelie_ALBERT. Could you please move all your data there? Thanks.

---

<div class="post-metadata">

**Author:** ![Aurelie\_ALBERT](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/aurelie_albert/32/210_2.png) [@Aurelie\_ALBERT](https://discourse.pangeo.io/u/Aurelie_ALBERT)\
**Post date:** [December 3, 2019, 8:29am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/10 "2019-12-03T08:29:11Z")

</div>

Thanks @rabernat I’m moving the data today !

---

<div class="post-metadata">

**Author:** ![Aurelie\_ALBERT](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/aurelie_albert/32/210_2.png) [@Aurelie\_ALBERT](https://discourse.pangeo.io/u/Aurelie_ALBERT)\
**Post date:** [December 3, 2019, 9:00am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/11 "2019-12-03T09:00:19Z")

</div>

Sorry @rabernat, there seems to be an issue with the pangeo-meom bucket, I get the error “BadRequestException: 400 Bucket is requester pays bucket but no user project provided.” each time I want to upload something or just “gsutil ls -a gs://pangeo-meom”. It must be linked with the parameter requester pays of the bucket but I cannot see it.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [December 3, 2019, 2:59pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/12 "2019-12-03T14:59:55Z")

</div>

I think you need to specify the user project now: `gsutil -u pangeo-181919 ...`

[https://cloud.google.com/storage/docs/using-requester-pays#using](https://cloud.google.com/storage/docs/using-requester-pays#using)

---

<div class="post-metadata">

**Author:** ![Aurelie\_ALBERT](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/aurelie_albert/32/210_2.png) [@Aurelie\_ALBERT](https://discourse.pangeo.io/u/Aurelie_ALBERT)\
**Post date:** [December 4, 2019, 8:32am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/13 "2019-12-04T08:32:25Z")

</div>

Ok, now I get “AccessDeniedException: 403 [auraoupa@gmail.com](mailto:auraoupa@gmail.com) does not have serviceusage.services.use access to project 464800473488.”

---

<div class="post-metadata">

**Author:** ![davidbrochart](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/davidbrochart/32/36_2.png) [@davidbrochart](https://discourse.pangeo.io/u/davidbrochart)\
**Post date:** [December 11, 2019, 4:32pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/14 "2019-12-11T16:32:02Z")

</div>

I’m going to work on uploading the GPM data again. So the gpm\_imerg data on pangeo-data can be deleted. Where should I upload now?

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [December 12, 2019, 2:35am UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/15 "2019-12-12T02:35:04Z")

</div>

@charlesbluca - could you assign a bucket to @davidbrochart?

---

<div class="post-metadata">

**Author:** ![kthyng](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/kthyng/32/19_2.png) [@kthyng](https://discourse.pangeo.io/u/kthyng)\
**Post date:** [December 17, 2019, 2:43pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/16 "2019-12-17T14:43:10Z")

</div>

Hi @rabernat and @jhamman! I am interested in getting a working example case available online for common tasks my group and interested parties want to do with our ROMS output. I checked out your data catalog and I don’t see model output in the ocean subsection. My hope is that we could store some output with Pangeo so that we could then use it in a Pangeo binder notebook for demonstration purposes. Somewhere between 10 and 100GB would be nice, and we could provide the output in zarr files if that is preferred. Maybe this is bad timing since you are ratcheting the data storage down but thought I’d see.

Alternatively, is there another good approach to have a working example case available online? I tried to use an hvplot with model output over the internet and it didn’t work, so it seems like even for demo purposes, the data and the analysis need to be co-located.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [December 17, 2019, 9:43pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/17 "2019-12-17T21:43:27Z")

</div>

Thanks for your post @kthyng! And welcome to the forum!

As you can probably tell, the cloud data catalog is currently very much a work in progress. The organization of the catalog has evolved organically, and we’re happy to see it continue to do so. There are many ocean model datasets, but not a specific section for models. If you have a suggestion for how to re-organize things, please feel free to propose it over in the github repo:

> **[pangeo-data/pangeo-datastore](https://github.com/pangeo-data/pangeo-datastore)**
>
> Pangeo Cloud Datastore. Contribute to pangeo-data/pangeo-datastore development by creating an account on GitHub.

In the long term, we would like to see a federation of data providers, rather than having Pangeo own all the data. (But still have this data be accessible through a single catalog.) Part of that process means getting people used to managing their own cloud storage.

In the case of your ROMS data, it costs very little. $100 GB of storage costs roughly $25 per year.

> **[Google Cloud Platform Pricing Calculator](https://cloud.google.com/products/calculator/#id=723990aa-7c9f-4f76-b680-9cc417b56777)**
>
> Create your own Custom Price Quote for the products offered through Google Cloud Platform based on number, usage, and power of servers

We could easily absorb this cost and create a new bucket for your data. But more than a financial cost, this creates a maintenance cost. I’d like to propose an alternative: _could you host the data under your own google cloud account?_ When you sign up for Google cloud, you’ll get $300 worth of free credits, enough to maintain your dataset for 12 years. (To avoid egress costs, put the bucket in “requester pays” mode.) This would give you total ownership over your own data and also teach you something about cloud storage.

Once you have the data uploaded, you could then make a PR to add it to the Pangeo catalog.

How does this sound? 😄

---

<div class="post-metadata">

**Author:** ![kthyng](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/kthyng/32/19_2.png) [@kthyng](https://discourse.pangeo.io/u/kthyng)\
**Post date:** [December 17, 2019, 10:07pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/18 "2019-12-17T22:07:01Z")

</div>

Yes, that sounds fair enough @rabernat. Just signed up for a bucket the other day in fact.

How do you see the other side of things working — the Pangeo binder side? Is it still possible to contribute a notebook to be able to run through Pangeo binder?

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [December 18, 2019, 2:17pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/19 "2019-12-18T14:17:31Z")

</div>

> [@kthyng](#):
>
> How do you see the other side of things working — the Pangeo binder side? Is it still possible to contribute a notebook to be able to run through Pangeo binder?

Absolutely! Anyone can do this now already, with no special permissions from us. Just go to [https://binder.pangeo.io/](https://binder.pangeo.io/).

Let us know if you get stuck on anything.

---

<div class="post-metadata">

**Author:** ![kthyng](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/kthyng/32/19_2.png) [@kthyng](https://discourse.pangeo.io/u/kthyng)\
**Post date:** [December 19, 2019, 7:55pm UTC](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353/20 "2019-12-19T19:55:42Z")

</div>

Awesome, thanks! I’ll be working on it.

[Next page](https://discourse.pangeo.io/t/cleaning-out-the-pangeo-data-google-cloud-storage-bucket/353.md?page=2)