# Delete access to Google Cloud Storage object

**URL:** <https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182>\
**Category:** Pangeo Cloud Support\
**Created:** [February 16, 2022, 8:08pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182 "2022-02-16T20:08:34Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![jdldeauna](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jdldeauna/32/1155_2.png) [@jdldeauna](https://discourse.pangeo.io/u/jdldeauna)\
**Post date:** [February 16, 2022, 8:08pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/1 "2022-02-16T20:08:34Z")

</div>

Hi! I’m trying to run the [rechunker Cloud example](https://rechunker.readthedocs.io/en/latest/tutorial.html#Cloud-Example):

```python
url = 'gs://pangeo-cmems-duacs'
gcs = gcsfs.GCSFileSystem(requester_pays=True)
source_store = gcs.get_mapper(url)

group = zarr.open_consolidated(source_store, mode='r')
source_array = group['sla']

max_mem = '1GB'
target_chunks = (8901, 72, 72)

scratch_path = os.environ['PANGEO_SCRATCH']

store_tmp = gcs.get_mapper(f'{scratch_path}/jdldeauna/rechunker_demo/temp_data_8.zarr' )
store_target = gcs.get_mapper(f'{scratch_path}/jdldeauna/rechunker_demo/target_data_8.zarr')

r = rechunk(source_array, target_chunks, max_mem,
                      store_target, temp_store=store_tmp)

```

However, executing r produces the following error:

```nohighlight
result = r.execute()

OSError: Forbidden: https://storage.googleapis.com/upload/storage/v1/b/pangeo-integration-te-3eea-prod-scratch-bucket/o
prod-user-sa@pangeo-integration-te-3eea.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.

```

Is it recommended to have a [Google Service Account](https://cloud.google.com/iam/docs/service-accounts) to be able to work with rechunker? I would appreciate any suggestions, thank you so much!

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [February 17, 2022, 2:07pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/2 "2022-02-17T14:07:10Z")

</div>

I just confirmed this. Here is a minimal reproducer

```python
import os
import fsspec
import gcsfs

with fsspec.open(os.environ['PANGEO_SCRATCH'] + '/test', mode='w') as fp:
    fp.write('foobar')

fs = gcsfs.GCSFileSystem()
fs.ls(os.environ['PANGEO_SCRATCH'])

fs.rm(os.environ['PANGEO_SCRATCH'] + '/test')

```

Gives

```nohighlight
prod-user-sa@pangeo-integration-te-3eea.iam.gserviceaccount.com does not have 
storage.objects.delete access to the Google Cloud Storage object.

```

I will have someone from 2i2c look into changing the permissions.

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [February 17, 2022, 3:11pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/3 "2022-02-17T15:11:11Z")

</div>

Quick note: this _might_ have been intentional. Recall that we don’t have user “namespaces” in the scratch bucket, and so granting delete access will let everyone delete everyone else’s files in the scratch bucket. That might be an acceptable tradeoff for a group of trusted users (you can already read everyone else’s scratch files)

---

<div class="post-metadata">

**Author:** ![jdldeauna](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jdldeauna/32/1155_2.png) [@jdldeauna](https://discourse.pangeo.io/u/jdldeauna)\
**Post date:** [February 17, 2022, 5:34pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/4 "2022-02-17T17:34:31Z")

</div>

Thanks @rabernat for looking into this! @TomAugspurger would it be possible to just have write access to the scratch bucket? I understand that files are already deleted every 7 days, so I don’t really need delete access. I’m not sure though why executing a rechunk requires delete access according to the error.

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [February 17, 2022, 6:18pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/5 "2022-02-17T18:18:09Z")

</div>

I think that specific error is from rechunker trying to write to a clean directory. In the meantime, you might try setting `store_target` to a non-existent directory, like `store_target = gcs.get_mapper(f'{scratch_path}/jdldeauna/rechunker_demo/target_data_9.zarr')`

---

<div class="post-metadata">

**Author:** ![jdldeauna](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jdldeauna/32/1155_2.png) [@jdldeauna](https://discourse.pangeo.io/u/jdldeauna)\
**Post date:** [February 18, 2022, 7:34pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/6 "2022-02-18T19:34:19Z")

</div>

Unfortunately even with changing the directory name for temp / target, the same error appears once the recheck is executed ☹

```nohighlight
result = r.execute()

OSError: Forbidden: https://storage.googleapis.com/upload/storage/v1/b/pangeo-integration-te-3eea-prod-scratch-bucket/o
prod-user-sa@pangeo-integration-te-3eea.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.

```

---

<div class="post-metadata">

**Author:** ![choldgraf](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/choldgraf/32/1061_2.png) [@choldgraf](https://discourse.pangeo.io/u/choldgraf)\
**Post date:** [February 21, 2022, 9:38pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/7 "2022-02-21T21:38:33Z")

</div>

Hey all - it sounds like there was a bit of uncertainty on whether this was “intended behavior” or not. As others mentioned, right now people can `read/write` to scratch, but they can’t `delete`. Can we get a confirmation that Pangeo community wants to give everybody the ability to delete as well?

---

<div class="post-metadata">

**Author:** ![jdldeauna](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jdldeauna/32/1155_2.png) [@jdldeauna](https://discourse.pangeo.io/u/jdldeauna)\
**Post date:** [February 21, 2022, 10:56pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/8 "2022-02-21T22:56:01Z")

</div>

Hi! I’m really sorry for the confusion, the error appears to be related to delete access, but it appears even when I’m only trying to write to my scratch bucket. When I run part of Ryan’s reproducer for example:

```python
import os
import fsspec
import gcsfs

with fsspec.open(os.environ['PANGEO_SCRATCH'] + '/test', mode='w') as fp:
    fp.write('foobar')

```

A similar error appears:

```python
OSError: Forbidden: https://storage.googleapis.com/upload/storage/v1/b/pangeo-integration-te-3eea-prod-scratch-bucket/o?uploadType=resumable&upload_id=ADPycdt_n7O4P4l-dNXBz4srYjyQv-LDHsLvnGnAV1LMvFNMzzo7wtkhJI7JOGRbHnvCPVsY3dMKPWSAdF0yAuPCzX8
prod-user-sa@pangeo-integration-te-3eea.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.

```

---

<div class="post-metadata">

**Author:** ![jbusecke](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jbusecke/32/118_2.png) [@jbusecke](https://discourse.pangeo.io/u/jbusecke)\
**Post date:** [February 22, 2022, 12:51am UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/9 "2022-02-22T00:51:46Z")

</div>

I personally would love to get `delete` access to the pangeo scratch bucket, for precisely that kind of processing @jdldeauna is doing here.  
I think it is a very common pattern to rechunk a large dataset, and then derive/save some much smaller output, which does not have to live on the scratch bucket.

@jdldeauna I think what you are trying to do is overwrite something, which AFAIK actually deletes first and then writes again, and thus needs delete rights. I have encountered that in the past.

I assume there is no way to allow some sort of permission, so that a user can only delete their own data?

---

<div class="post-metadata">

**Author:** ![jdldeauna](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/jdldeauna/32/1155_2.png) [@jdldeauna](https://discourse.pangeo.io/u/jdldeauna)\
**Post date:** [February 22, 2022, 3:02am UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/10 "2022-02-22T03:02:52Z")

</div>

Oh I see, if I change `with fsspec.open(os.environ['PANGEO_SCRATCH'] + '/test', mode='w') as fp:` to `'/test2'` I’m able to write the file, sorry I misunderstood that. Similarly, going back to the rechunker example, I was just changing the filename (e.g., `temp_data_8.zarr` to `temp_data_9.zarr`) when I should have also changed the folder name (e.g., `rechunker_demo`) to avoid overwriting previous data. As for the `delete` permissions I’m fine with however the community decides @choldgraf, but it might be nice to have personal access, as suggested by @jbusecke. Thank you!

---

<div class="post-metadata">

**Author:** ![sgibson91](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/sgibson91/32/1051_2.png) [@sgibson91](https://discourse.pangeo.io/u/sgibson91)\
**Post date:** [February 22, 2022, 1:59pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/11 "2022-02-22T13:59:16Z")

</div>

Hey all,

I believe object storage works a little bit like `git commit`, so `read/write` permissions allow you overwrite essentially by _creating a new file with the changes_. This is why `delete` is a separate permission because it can also mean _delete all versions of a file_.

> [@jbusecke](#):
>
> I assume there is no way to allow some sort of permission, so that a user can only delete their own data?

Unfortunately, this is true. However, I will enable the `delete` permission now as this is an experienced community.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [February 22, 2022, 3:00pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/12 "2022-02-22T15:00:49Z")

</div>

Thanks so much Sarah for your help! 🙏

Just noting that finding a good general solution to this problem–providing private “scratch” storage to cloud Jupyter users–is an important and challenging DevOps problem that we have been discussing in Pangeo for many years. The central challenge is that, within the Kubernetes cluster where the hub runs, there is no mapping between hub identity (e.g. the username you log into the hub with, usually from GitHub), and a unique cloud-provider identity. If there were such a mapping, we could just create a bucket for each hub users. But as is, all hub users look identical to the cloud-provider. So we have no choice but to provide uniform global access to the scratch bucket for all users.

If any DevOps engineers are reading this and would like to work towards a better solution, please jump right in!

---

<div class="post-metadata">

**Author:** ![sgibson91](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/sgibson91/32/1051_2.png) [@sgibson91](https://discourse.pangeo.io/u/sgibson91)\
**Post date:** [February 22, 2022, 3:13pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/13 "2022-02-22T15:13:46Z")

</div>

I believe this is now working (screenshot is a test on staging, but I have propagated that change to prod as well)

 ![Screenshot 2022-02-22 at 15.10.14](https://canada1.discourse-cdn.com/flex030/uploads/pangeo/original/2X/8/87b753e4589cd3c736292bcd57713e488f2fea50.png)

---

<div class="post-metadata">

**Author:** ![TomAugspurger](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/tomaugspurger/32/21_2.png) [@TomAugspurger](https://discourse.pangeo.io/u/TomAugspurger)\
**Post date:** [February 22, 2022, 4:03pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/14 "2022-02-22T16:03:30Z")

</div>

> [@rabernat](#):
>
> If any DevOps engineers are reading this and would like to work towards a better solution, please jump right in!

We’re in the design stages for something similar on Azure (call it a “user-data” service). It’s likely that the concepts will generalize to other clouds. Our requirements are

1. Users can only view / modify only their own data.
2. The system is able to enforce some kind of quota on bytes stored per user.
3. When bytes are actually being written to / read from blob storage, we don’t want anything in between the user and the Blob Storage service.

First, you’ll need some sort of identity system. pangeo-cloud could piggyback on JupyterHub or use auth0. All requests to the user-data service must be authenticated.

For uploading data, users will make requests to the user data service, requesting permission to write a specific number of bytes to a specific key. The service will verify that this is OK (the user hasn’t exceeded their quota, for example), and will issue a [SAS token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview?toc=/azure/storage/blobs/toc.json) that can only write to that specific key. The user can upload the data using their normal means (fsspec/adlfs, or `azure.storage.blob`, etc.)

After it’s written, the user-data service will verify that the write is OK (e.g. isn’t larger than was requested) using [Azure’s event grid](https://docs.microsoft.com/en-us/azure/event-grid/overview). If it’s too large, we’ll delete it and somehow notify the user.

Reading specific keys is pretty similar. Users request permission to read a key and get a SAS token. Listing “directories” is more challenging, because Azure Blob Storage doesn’t have a built-in concept of SAS-tokens that are limited to prefixes. We’re still figuring that out.

So it’s yet another service to run, and pretty complicated to what pangeo-cloud has today (and it’s purely theoretical right now 😄) but we’ll post details if and when it becomes a reality.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [February 22, 2022, 4:10pm UTC](https://discourse.pangeo.io/t/delete-access-to-google-cloud-storage-object/2182/15 "2022-02-22T16:10:30Z")

</div>

Wait Tom you’re a DevOps engineer? I thought you were an oceanographer now! 😆

But seriously, this sounds very cool.