# Binder.pangeo.io shut down due to crypto mining

**URL:** <https://discourse.pangeo.io/t/binder-pangeo-io-shut-down-due-to-crypto-mining/2007>\
**Category:** Cloud\
**Created:** [December 8, 2021, 2:40pm UTC](https://discourse.pangeo.io/t/binder-pangeo-io-shut-down-due-to-crypto-mining/2007 "2021-12-08T14:40:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [December 8, 2021, 2:40pm UTC](https://discourse.pangeo.io/t/binder-pangeo-io-shut-down-due-to-crypto-mining/2007/1 "2021-12-08T14:40:57Z")

</div>

Details here.

> <https://github.com/pangeo-data/pangeo-binder/issues/195>
>
> A few weeks ago we started seeing these notices from Google CLoud
> 
> Dear Develo…per,
> 
> Our systems identified that your Google Cloud Platform / API Project ID
> pangeo (id: pangeo-181919) may have been compromised and used for
> cryptocurrency mining.
> 
> \<details\>
> 
> This activity was detected as originating from IP 34.134.139.27,
> 35.202.205.80, 34.122.114.163, 34.132.191.152, 35.202.175.45,
> 34.133.126.79, 34.123.101.98, 35.193.3.130, 34.132.119.240, 34.71.241.100,
> 34.67.130.14, 34.71.90.24, 34.135.253.5, 34.72.61.75, 35.222.246.237,
> 34.134.57.213, 34.133.118.37, 34.133.150.175, 34.67.43.143, 34.68.227.209,
> 35.232.249.88, 34.132.212.244, 34.132.10.154, 35.184.190.167, 35.224.43.11,
> 35.184.16.35, 34.72.110.84, 34.68.128.37, 104.154.53.176, 34.135.132.226,
> 34.71.220.222, 34.121.209.232, 104.154.77.62, 34.132.152.105,
> 34.136.19.208, 35.238.176.255, 35.192.104.29, 35.232.158.48,
> 34.123.200.254, 35.193.162.116, 34.70.19.67, 35.202.170.215, 35.226.59.255,
> 34.132.160.123, 35.184.88.34, 34.68.28.248, 34.67.150.235 and VM ID
> 7204308034468923890:us-central1-b 6154726994941753177:us-central1-b
> 23495819365936574:us-central1-b 871060581044631402:us-central1-b
> 9056366196714287444:us-central1-b 2078833108693727082:us-central1-b
> 2379172320502788252:us-central1-b 2033074746165521335:us-central1-b
> 6614086151114555388:us-central1-b 6481841045759038865:us-central1-b
> 8060108563441484120:us-central1-b 1930093761258502077:us-central1-b
> 6917237063263891611:us-central1-b 1769396228438798560:us-central1-b
> 1152664557098022744:us-central1-b 1368015935306407708:us-central1-b
> 8470523882921731892:us-central1-b 3453670015797211952:us-central1-b
> 7477268612177377130:us-central1-b 4230094429538660508:us-central1-b
> 7516664448808039216:us-central1-b 6658613447166542524:us-central1-b
> 2719659052643825153:us-central1-b 9025766762343476985:us-central1-b
> 3824956035727274209:us-central1-b 3042387495175264580:us-central1-b
> 2566597537884714993:us-central1-b 4099334863176513368:us-central1-b
> 1453326668843364089:us-central1-b 6008087590161573013:us-central1-b
> 4157158282760470131:us-central1-b 3620565006036418163:us-central1-b
> 2777884811879213922:us-central1-b 4826068438038551749:us-central1-b
> 2366113575518516063:us-central1-b 7870704371158398748:us-central1-b
> 3614756977596609295:us-central1-b 3603490415090774185:us-central1-b
> 1726988374193171917:us-central1-b 6533022912599996248:us-central1-b
> 85662215390327083:us-central1-b 8669231498353883121:us-central1-b
> 4296773947132917694:us-central1-b 7896205443125692539:us-central1-b
> 7867110535798659555:us-central1-b 1955607456134639840:us-central1-b
> 6190384449088303272:us-central1-b to destination IP 51.79.251.11 on remote
> port 3300&nbsp;between 2021-12-07 01:59 and 2021-12-07 02:25 (Pacific
> Time), though it may still be ongoing.
> 
> We recommend that you review this activity to determine if it is intended.
> Cryptocurrency mining is often an indication of the use of fraudulent
> accounts and payment instruments, and we require verification in order to
> mine cryptocurrency on our platform.&nbsp;Additional information is
> available in the Cloud Security Help
> Center(support.google.com/cloud/answer/6262505).
> 
> If you believe your project has been compromised, we recommend that you
> secure all your instances
> (https://support.google.com/cloud/answer/6262505), which may require
> uninstalling and then re-installing your project.
> 
> To better protect your organization from misconfiguration and access the
> best of Google's threat detection, you may consider enabling Security
> Command Center (SCC) for your organization. To learn more about SCC visit
> https://cloud.google.com/security-command-center.
> 
> Once&nbsp;you have fixed the issue, please respond to this email. If the
> behavior is intentional, please explain so that we do not ping you again
> for this activity. Please do not hesitate to reach out to us if you have
> questions.
> 
> \</details\>
> 
> Crypto mining is a common problem on binder deployments, and it has finally hit us.
> 
> I ignored it for a while. We currently have no sysadmin for the binder cluster. It is running totally unsupervised. However, I recently checked the logs and noticed a huge spike in usage:
> 
> !\[image\](https://user-images.githubusercontent.com/1197350/145225214-19714671-24dd-44b9-af39-4cbc2ba759d4.png)
> 
> The binder has been in maxed-out state for quite a while, and is on track to cost thousands of more dollars per month than we are used to.
> 
> \### Resolution
> 
> I needed to try to shut down the binder as fast as possible. Unfortunately, my kubernetes / helm skills are very rusty. Here's what I tried:
> 
> First I updated my local gcloud and helm to latest versions (haven't touched either in over a year). Then
> 
> \`\`\`bash
> gcloud auth login
> gcloud container clusters get-credentials binder
> \`\`\`
> 
> Then I tried helm
> \`\`\`bash
> $ helm init
> WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
> WARNING: You should switch to "https://charts.helm.sh/stable"
> $HELM\_HOME has been configured at /Users/rpa/.helm.
> Warning: Tiller is already installed in the cluster.
> (Use --client-only to suppress this message, or --upgrade to upgrade Tiller to the current version.)
> 
> $ helm status
> WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
> WARNING: You should switch to "https://charts.helm.sh/stable"
> Error: could not find a ready tiller pod
> \`\`\`
> 
> I couldn't get anything useful out of helm, so I gave up on it.
> 
> Then I went to kubernetes
> \`\`\`python
> kubectl delete --all pods --namespace prod
> kubectl delete --all pods --namespace staging
> \`\`\`
> 
> This had no apparent effect.
> 
> So then I went to https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/binder/nodes?project=pangeo-181919 and manually resized all the node pools to zero. That seemed to work.
> 
> So the binder is currently completely broken and unusable. 😞 It would be nice to at least get a landing page up at binder.pangeo.io that explains the situation. I am worried that this will affect activities planned for AGU, but I'm not sure.
> 
> I think our best hope for reviving our binder would be when 2i2c can take on this deployment, but that likely won't happen until spring.

This is a very unfortunate situation.

---

<div class="post-metadata">

**Author:** ![rabernat](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pangeo.io/rabernat/32/22_2.png) [@rabernat](https://discourse.pangeo.io/u/rabernat)\
**Post date:** [September 16, 2022, 4:33pm UTC](https://discourse.pangeo.io/t/binder-pangeo-io-shut-down-due-to-crypto-mining/2007/2 "2022-09-16T16:33:38Z")

</div>

Just an update that we recently had a meeting with @jmunroe, @sgibson91, @yuvipanda, and Damián Avila of 2i2c to sort through the future for Pangeo binder. There are some notes here if anyone is interested:

> **[Pangeo Binder Meeting](https://docs.google.com/document/d/1P8VE6ptmPAEFfrSO9GRma7JHdZS4XBC-EN32J2Ka5hY/edit)**
>
> Pangeo Binder Meeting - Date: Friday 16th September 2022 - Time: 11:30 AM ET Attendees: Ryan Abernathey James Munroe Sarah Gibson Damian Avila Yuvi Panda Agenda: Review overall Pangeo &lt;&gt; 2i2c collab Active contracts Moore Foundation LEAP...