Currently running Rancher 2.2 on an Openstack provider. Have managed to get cluster auto-provisioning working along with some other nice things. However, and this might be more of a K8S thing, but I need to now look at HPA, or scaling pods out automatically based on load.
Has anyone successfully done this in Rancher? Or even in K8S at all? Looking for any advice anyone can give. Thanks.
So continuing my own work into this. I’ve managed to get it slightly working in that if I set a min number of pods it scales up to the min number. However, the metric I want to use current_connections, which documentation and walkthroughs says is a valid metric fails. I get the following from kubectl describe hpa
Type Reason Age From Message
Normal SuccessfulRescale 2m49s horizontal-pod-autoscaler New size: 2; reason: Current number of replicas below Spec.MinReplicas
Warning FailedGetPodsMetric 4s (x11 over 2m34s) horizontal-pod-autoscaler unable to get metric current_connections: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
Warning FailedComputeMetricsReplicas 4s (x11 over 2m34s) horizontal-pod-autoscaler failed to get object metric value: unable to get metric current_connections: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
My HPA is setup as so:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: mydeployment-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: mydeployment
minReplicas: 2
maxReplicas: 10
metrics:
We have HPA examples on https://rancher.com/docs/rancher/v2.x/en/k8s-in-rancher/horitzontal-pod-autoscaler/ and you can check out https://gist.github.com/superseb/4147cbbe432d11b63e3a4432d12df2b7 for some walkthrough and debugging steps
I find it annoying when replies simply point to the docs, I’ve looked at the docs.
With regards to your first post, I’m using Rancher 2.2 so according to the docs, it has the metrics-server already setup and configured. So all that’s needed is to push the actual HPA with the workload. And I can legitimately see that pod, but even a basic HPA setup to use CPU monitoring, taken from valid examples does not work.
The second example is worthless with 2.2 which has prometheus built in, you try to run the steps and some fail as they are already present.
Interested in real world examples of someone successfully doing HPA on Rancher specifically. Because as things stand right now, it is almost impossible. Only thing that’s slightly out of the ordinary is that I’m deployed on Openstack, but cluster deployments, and even auto-creation of loadbalancers work flawlessly. Only thing I’m having issues is that the metrics and HPA is broken or has some specifics needed that are hidden.
So doing HPA via CPU usage is working for me now, what wasn’t specified when I did my research, but makes sense is that before you can use HPA to scale out, you need to place limits on the POD/Deployment. Once I placed limits, and then threw some load against the site it worked as needed. Scaled out to 8 pods, then back to the min 2 when I stopped my testing.
Now to work on the custom metrics stuff.