Us-central1 pangeo hub down?

1

Hi there,

Whilst it was working fine yesterday, today I cannot connect at all to https://us-central1-b.gcp.pangeo.io/

Does anyone know what is going on here, and how I may access this? I have some files on it I’d really like to retrieve!

Thanks in advance,

Andrew

2

+1,

I thought it was something with my connection or system, but have tried different networks without success.

3

@rabernat warns me that it may simply have run out of funds. However, I think that an at risk or other warning message to users would have been warranted before pulling the plug (if that is what has happened). I’ve contacted their support services and await a response, I suggest you do the same!

4

Thanks,

I am not sure what support services you mean?

5

Sorry, I’ve only learnt this today too! 2i2c are the hosts of the platform. I lodged my support ticket here: Get support — Hub Service Guide

1 Like
6

Hi all,

I’ll jump in dual role as being part of 2i2c (the org providing the support to the service) and a member of the pangeo steering council.

2i2c is aware that as of last night that all the pangeo hubs have gone done. As well, the 2i2c engineers appear to now no longer have access to any of the GCP accounts that were used to manage this resource.

If there is data that need to be recovered or migrated we need to have a discussion at some level.

More to come.

James

1 Like
7

Thanks @jmunroe I would very much like to get at the data there, so please do keep us in the loop. I’m not sure what you mean specifically by GCP account? Do you mean they don’t know which username/login to use to bring it back?

8

Thanks @jmunroe , I also have a repository on this hub in use. :sob: Fortunately I have stored all changes elsewhere, but it would be good to regain access. Do you think the hub will be up again?

9

I am referring to the Google Cloud Platform engineering-level accounts that administer the kubernetes cluster that https://us-central1-b.gcp.pangeo.io/ lives within. While access is presently disabled – the question at the moment is who is able to see those cloud resources so we can begin to figure out how/if to recover.

I see two immediate needs here:

  1. Self-identification of user accounts (e.g. github ids) that have high-value data or work on us-central1-b.gcp.pangeo.io. If your data is stored as a repo on GitHub or replicated elsewhere or is now longer relevant that is much different than your data being part of active research project for which no recent back up exists. Understanding who is critically impacted by this outage is important to me. If there is something we can do to help those critically impacted users, we, as a Pangeo Community, should do it.
  2. A conversation needs to occur between 2i2c and Columbia IT (the GCP resources in question where managed under Columbia University with accounts created for 2i2c engineer).

I am feeling some conflict-of-interest being in a dual-role as oth working at 2i2c and being a member of the Pangeo community. I don’t think it is appropriate that I attempt to wear both hats in the conversation. Is there anyone else on the Pangeo Steering Council who is willing to have a conversation on behalf the Pangeo community so we can work towards a resolution? (I would feel very awkward having a meeting with myself :sweat_smile:)

After there is more clarity on what the options I hope we will be able we can provide more clarity on the https://us-central1-b.gcp.pangeo.io hub to the Pangeo community.

James

1 Like
10

Thanks for the clarity. I’m not a member of the council unfortunately, so can’t step into that role, but I hope someone is found soon. For reference, I am severely impacted by this. I’ve got a lot of code that doesn’t have recent reversions there, as well as a non-trivial amount of data output that is critical to ongoing research papers in active development.

11

Okay, thanks @jmunroe! Hopefully someone from the Pangeo Steering Council could join the conversation.
The repo I mention is very much part of an active research project, so we would love it if further use of the hub would be possible. (We were not aware of this downtime.)

  • I understand it so that it is just the access to the hubs that is shut down, correct? So that data in adjacent cloud-buckets is still there? We do have lots of results/output from several computations, stored, not on the hub itself, but on a project specific bucket, that we need to keep. Hopefully that is not part of this downtime…
12

I have some old access to a GCP project named pangeo with

  • project number: 464800473488
  • project id: pangeo-181919

Is that where these resources are at? I’m not too familiar with the GCP console but I don’t immediately see any VMs or Kubernetes Clusters that look like this Hub.

13

Thanks Tom, but no that’s not it. That is the old GCP account we migrated from when 2i2c took over stewardship.

ETA: I have contacted Columbia IT to begin a discussion and try to understand why 2i2c engineers can no longer see the GCP project.

1 Like
14

Thank you @sgibson91

15

Hi Folks! This hub has been on autopilot for a long time. From an institutional point of view, this hub is “owned” by Columbia University. (That’s where the GCP project lives.) However, I, the PI for the award that was funding it, no longer work there as of this past summer! So the hub may be in a bit of an orphaned state.

We’ll see if we can get it back online long enough for folks to download their work. But without a new faculty champion at Columbia, it will be a temporary solution at best.

16

@rabernat thanks! Long enough to download would be a huge relief.

17

@jmunroe @rabernat any progress on this issue? I don’t want to nag on this, but it’d be great to go into the weekend knowing how much I need to stress about my data and paper progress!

18

Unfortunately, we are still waiting for information ourselves.

19

A +1, in case this information helps:

We also have the same issue, most of the data we use is stored on a bucket I believe adjacent to the local hub storage.

Thanks to @slunav and @annefou I cloned the repo to Pangeo@EOSC today, and from there we are able to list our objects stored in the US Central bucket using the file system interface to GCS. Luckily the data is present. However accessing zarr-files (or any other file) is not possible and returns:

OSError: Forbidden: https://storage.googleapis.com/download/storage/v1/b/pangeo-argo-eke/o/testfolder_112%2FEKE%2Fglobal_timemean.zarr%2F.zmetadata?alt=media
The billing account for the owning project is disabled in state absent

Which perhaps supports the warning from you and @rabernat, about simply running out of funds being the issue? We hope there is an easy way to regain access to download the data.

And if there is a way to refund the same hub, that would obviously be very helpful for our project which has some amount of computations left, planned to be completed this month… But we also understand that funding does not last forever. (Our “pipeline” has, not surprisingly, been quite tailored for being run on US Central hub (the amount of workers, and RAM per available worker)).

20

Thanks for this @ofk123 . I’m not familiar enough with the setup to know if this represents a general solution for other users or not? Would this be a technique that could be used to access files stored on the us-central1 pangeo hub for all of us? If so, I’d love to know more about how you did it in practice.

I’d also very much appreciate it if the hub were refunded, though at the moment I’ll definitely settle for just getting access to the files presently there.