A few weeks ago we started seeing these notices from Google CLoud
Dear Develo…per,
Our systems identified that your Google Cloud Platform / API Project ID
pangeo (id: pangeo-181919) may have been compromised and used for
cryptocurrency mining.
<details>
This activity was detected as originating from IP 34.134.139.27,
35.202.205.80, 34.122.114.163, 34.132.191.152, 35.202.175.45,
34.133.126.79, 34.123.101.98, 35.193.3.130, 34.132.119.240, 34.71.241.100,
34.67.130.14, 34.71.90.24, 34.135.253.5, 34.72.61.75, 35.222.246.237,
34.134.57.213, 34.133.118.37, 34.133.150.175, 34.67.43.143, 34.68.227.209,
35.232.249.88, 34.132.212.244, 34.132.10.154, 35.184.190.167, 35.224.43.11,
35.184.16.35, 34.72.110.84, 34.68.128.37, 104.154.53.176, 34.135.132.226,
34.71.220.222, 34.121.209.232, 104.154.77.62, 34.132.152.105,
34.136.19.208, 35.238.176.255, 35.192.104.29, 35.232.158.48,
34.123.200.254, 35.193.162.116, 34.70.19.67, 35.202.170.215, 35.226.59.255,
34.132.160.123, 35.184.88.34, 34.68.28.248, 34.67.150.235 and VM ID
7204308034468923890:us-central1-b 6154726994941753177:us-central1-b
23495819365936574:us-central1-b 871060581044631402:us-central1-b
9056366196714287444:us-central1-b 2078833108693727082:us-central1-b
2379172320502788252:us-central1-b 2033074746165521335:us-central1-b
6614086151114555388:us-central1-b 6481841045759038865:us-central1-b
8060108563441484120:us-central1-b 1930093761258502077:us-central1-b
6917237063263891611:us-central1-b 1769396228438798560:us-central1-b
1152664557098022744:us-central1-b 1368015935306407708:us-central1-b
8470523882921731892:us-central1-b 3453670015797211952:us-central1-b
7477268612177377130:us-central1-b 4230094429538660508:us-central1-b
7516664448808039216:us-central1-b 6658613447166542524:us-central1-b
2719659052643825153:us-central1-b 9025766762343476985:us-central1-b
3824956035727274209:us-central1-b 3042387495175264580:us-central1-b
2566597537884714993:us-central1-b 4099334863176513368:us-central1-b
1453326668843364089:us-central1-b 6008087590161573013:us-central1-b
4157158282760470131:us-central1-b 3620565006036418163:us-central1-b
2777884811879213922:us-central1-b 4826068438038551749:us-central1-b
2366113575518516063:us-central1-b 7870704371158398748:us-central1-b
3614756977596609295:us-central1-b 3603490415090774185:us-central1-b
1726988374193171917:us-central1-b 6533022912599996248:us-central1-b
85662215390327083:us-central1-b 8669231498353883121:us-central1-b
4296773947132917694:us-central1-b 7896205443125692539:us-central1-b
7867110535798659555:us-central1-b 1955607456134639840:us-central1-b
6190384449088303272:us-central1-b to destination IP 51.79.251.11 on remote
port 3300 between 2021-12-07 01:59 and 2021-12-07 02:25 (Pacific
Time), though it may still be ongoing.
We recommend that you review this activity to determine if it is intended.
Cryptocurrency mining is often an indication of the use of fraudulent
accounts and payment instruments, and we require verification in order to
mine cryptocurrency on our platform. Additional information is
available in the Cloud Security Help
Center(support.google.com/cloud/answer/6262505).
If you believe your project has been compromised, we recommend that you
secure all your instances
(https://support.google.com/cloud/answer/6262505), which may require
uninstalling and then re-installing your project.
To better protect your organization from misconfiguration and access the
best of Google's threat detection, you may consider enabling Security
Command Center (SCC) for your organization. To learn more about SCC visit
https://cloud.google.com/security-command-center.
Once you have fixed the issue, please respond to this email. If the
behavior is intentional, please explain so that we do not ping you again
for this activity. Please do not hesitate to reach out to us if you have
questions.
</details>
Crypto mining is a common problem on binder deployments, and it has finally hit us.
I ignored it for a while. We currently have no sysadmin for the binder cluster. It is running totally unsupervised. However, I recently checked the logs and noticed a huge spike in usage:
The binder has been in maxed-out state for quite a while, and is on track to cost thousands of more dollars per month than we are used to.
### Resolution
I needed to try to shut down the binder as fast as possible. Unfortunately, my kubernetes / helm skills are very rusty. Here's what I tried:
First I updated my local gcloud and helm to latest versions (haven't touched either in over a year). Then
```bash
gcloud auth login
gcloud container clusters get-credentials binder
```
Then I tried helm
```bash
$ helm init
WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
WARNING: You should switch to "https://charts.helm.sh/stable"
$HELM_HOME has been configured at /Users/rpa/.helm.
Warning: Tiller is already installed in the cluster.
(Use --client-only to suppress this message, or --upgrade to upgrade Tiller to the current version.)
$ helm status
WARNING: "kubernetes-charts.storage.googleapis.com" is deprecated for "stable" and will be deleted Nov. 13, 2020.
WARNING: You should switch to "https://charts.helm.sh/stable"
Error: could not find a ready tiller pod
```
I couldn't get anything useful out of helm, so I gave up on it.
Then I went to kubernetes
```python
kubectl delete --all pods --namespace prod
kubectl delete --all pods --namespace staging
```
This had no apparent effect.
So then I went to https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/binder/nodes?project=pangeo-181919 and manually resized all the node pools to zero. That seemed to work.
So the binder is currently completely broken and unusable. 😞 It would be nice to at least get a landing page up at binder.pangeo.io that explains the situation. I am worried that this will affect activities planned for AGU, but I'm not sure.
I think our best hope for reviving our binder would be when 2i2c can take on this deployment, but that likely won't happen until spring.