Us-central1 pangeo hub down?

41

Have we had any update on this, I’m still very keen to get to my data!

42

I’ve had to escalate this upwards for the time being.

43

Great, thanks for the quick response! Hopefully you get some replies soon! Thanks again for handling this for the community.

44

Thanks alot for the time you spend and spent @sgibson91 and Steeringcouncil,
Hopefully access will resolve soon.

45

Hi all, gently touching in again here. Have we heard anything from the escalation yet?

46

Hi all, we wonder the same. Any news regarding the status of the hub-access?

47

Hi again. Is there a possibility of requesting an update from the people you escalated this to please? I’m stuck in a terrible limbo of not knowing whether I should be starting the mammoth task of recreating my data, or doing something else useful whilst waiting for the connection to be restored. Thanks again for all your efforts on this!

48

I understand it is very frustrating. I’m deeply sorry for the disruption this has caused to peoples’ work. We are are still working with CUIT to attempt to regain access to the user data. Communication and iteration have been slow, but there is progress. We can’t promise anything and don’t know any timelines. I know it sucks to hear that.

49

Thanks Ryan, we really appreciate the work you’re all putting in here to save our data! Is there any indication of what the stumbling block is? Hopefully it can be resolved soon, and thanks again for this.

50

I’ll share one glimmer of hope. It looks like the filestore instance where user home directories are stored still exists. So hopefully the data have not been permanently deleted. However, we are still not able to access the data. Fortunately the 2i2c team is on the case and will hopefully have it resolved soon.

51

Thanks for the glimmer Ryan! Any update on getting access to that filestore?

52

Gentle bump on this one folks @rabernat @sgibson91

53

Hi All,

I’m here to share a very disappointing update. We finally got a response from Google. I’m quoting from CUIT:

Our billing provider received a response from Google today. They (Google) managed to restore the filestore. Unfortunately all the data was removed on November 14th. Google said the data is unrecoverable.

So all data in the cluster home directories have been lost.

I’m deeply sorry for this. 100% of the responsibility for this situation lies with me. While I didn’t foresee that things would have unfolded this way, ultimately I failed to make a good transition plan for what would happen to these Google Cloud resources after I left my position at Columbia.

If you are a student or research who has been impacted by this, and you’d like someone to explain to you PI why your project has been delayed, I can write a personal email explaining the situation and that it’s not your fault. Please just DM me.

To everyone else, I don’t know what to say…feeling full of shame and regret over how this turned out.

-Ryan

2 Likes
54

To clarify, only the filestore (hub home directories) has been lost.

All of the data in GCS buckets is still there. (But is long overdue for some cleaning.)

55

Hi Ryan.

Thanks for chasing this down. It’s a very disappointing outcome, but a salutary lesson (again) about the importance of backups, even in the cloud.

I’m pretty sad about the lost data (I was actively writing a paper with it), but at least now I know I can get started on recreating it without worrying that I’m wasting my time and my old data may reappear at some point.

Thanks again for your hard work on this, and sorry you got drawn back into something you’d thought to have left behind!

Now off to have a stiff drink I think…

56

Thanks @rabernat for the update,

I have learned from this as well. We use a GSC Bucket that I think you sat up early on, and have been syncing our code, so we have no losses from the homedirectory.

  • But is there an expiration date on the bucket as well?

We plan on moving data to a permanent location, so that our project takes the storage-cost onward.

For info, gcsfs.GCSFileSystem(token=token).info(bucketname) returns, among other things:

{'kind': 'storage#bucket', 
 'name': 'pangeo-argo-eke',
 'location': 'US-CENTRAL1',
 'storageClass': 'STANDARD',
 'timeCreated': '2021-01-07T17:40:07.103Z',
 'updated': '2021-01-07T20:25:44.733Z',
 'softDeletePolicy': {'retentionDurationSeconds': '604800',
  'effectiveTime': '2024-03-01T08:00:00.000Z'},
...

And thanks again for the work you and @steeringcouncil has put into this!

57

Hi Ryan,
Thanks for sharing the information. May I ask if the data on GCS buckets will stay there? If not, is there a specific timeline for how long they will be retained?
Also, thank you very much for maintaining this cloud service available for everyone for such a long time. I really appreciate your work.