I am trying to set up horizontal pod autoscaling with custom metrics in Rancher v2.5. I installed rancher-monitoring following instructions here. This installed prometheus-adapter in the cluster as well.
I updated the prometheus-adapter configMap to add my custom metric as shown below (truncated output for brevity):
➜ ~ kubectl -n cattle-monitoring-system get configmap rancher-monitoring-prometheus-adapter -o yaml
...
- seriesQuery: 'CoolGardenRequestListener_requests_age_milliseconds_max{namespace!="",pod!=""}'
resources:
template: <<.Resource>>
metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
name:
matches: ^(.*)
as: "coolgarden_request_age"
...
I ran the following to ensure this metric was available via k8s api:
➜ ~ kubectl get --raw "/k8s/clusters/c-t66dx/apis/custom.metrics.k8s.io/v1beta1/namespaces/coolgardener/pods/*/coolgarden_request_age" | jq
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/coolgardener/pods/%2A/coolgarden_request_age"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "coolgardener",
"name": "coolgardener-6f9cf4595-klrz9",
"apiVersion": "/v1"
},
"metricName": "coolgarden_request_age",
"timestamp": "2021-04-13T20:52:52Z",
"value": "19373868",
"selector": null
},
{
"describedObject": {
"kind": "Pod",
"namespace": "coolgardener",
"name": "coolgardener-6f9cf4595-v4xrt",
"apiVersion": "/v1"
},
"metricName": "coolgarden_request_age",
"timestamp": "2021-04-13T20:52:52Z",
"value": "78170499",
"selector": null
}
]
}
Then I proceeded to create an HPA using the following:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: coolgarden-hpa
namespace: coolgardener
spec:
maxReplicas: 3
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: coolgardener
metrics:
- type: Pods
pods:
metric:
name: coolgarden_request_age
target:
type: Value
averageValue: 3600k
However, the HPA errors out as shown below:
➜ ~ kubectl describe horizontalpodautoscaler coolgarden-hpa
Name: coolgarden-hpa
Namespace: coolgardener
Labels: cattle.io/creator=norman
Annotations: field.cattle.io/creatorId: u-x5rfkgkxtg
field.cattle.io/displayName: mc-hpa-1
CreationTimestamp: Tue, 13 Apr 2021 12:59:15 -0700
Reference: Deployment/coolgardener
Metrics: ( current / target )
"coolgarden_request_age" on pods: <unknown> / 3600k
Min replicas: 1
Max replicas: 3
Deployment pods: 2 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetPodsMetric the HPA was unable to compute the replica count: unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 57m (x13 over 60m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
Warning FailedGetPodsMetric 32s (x234 over 60m) horizontal-pod-autoscaler unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
The pods.custom.metrics.k8s.io *
address seems to be incorrect (based on kubectl --raw
output above. Not sure where this is being set and how this can be updated.
Anything I’m doing wrong here?