Unable to fetch metrics from custom metrics API in Rancher v2.5

Rancher 2.x
1

I am trying to set up horizontal pod autoscaling with custom metrics in Rancher v2.5. I installed rancher-monitoring following instructions here. This installed prometheus-adapter in the cluster as well.

I updated the prometheus-adapter configMap to add my custom metric as shown below (truncated output for brevity):

➜  ~ kubectl -n cattle-monitoring-system get configmap rancher-monitoring-prometheus-adapter -o yaml 
...
      - seriesQuery: 'CoolGardenRequestListener_requests_age_milliseconds_max{namespace!="",pod!=""}'
        resources:
          template: <<.Resource>>
        metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
        name:
          matches: ^(.*)
          as: "coolgarden_request_age"
...

I ran the following to ensure this metric was available via k8s api:

➜  ~ kubectl get --raw "/k8s/clusters/c-t66dx/apis/custom.metrics.k8s.io/v1beta1/namespaces/coolgardener/pods/*/coolgarden_request_age" | jq 
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/coolgardener/pods/%2A/coolgarden_request_age"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "coolgardener",
        "name": "coolgardener-6f9cf4595-klrz9",
        "apiVersion": "/v1"
      },
      "metricName": "coolgarden_request_age",
      "timestamp": "2021-04-13T20:52:52Z",
      "value": "19373868",
      "selector": null
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "coolgardener",
        "name": "coolgardener-6f9cf4595-v4xrt",
        "apiVersion": "/v1"
      },
      "metricName": "coolgarden_request_age",
      "timestamp": "2021-04-13T20:52:52Z",
      "value": "78170499",
      "selector": null
    }
  ]
}

Then I proceeded to create an HPA using the following:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: coolgarden-hpa
  namespace: coolgardener
spec:
  maxReplicas: 3
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coolgardener
  metrics:
    - type: Pods
      pods:
        metric:
          name: coolgarden_request_age
        target:
          type: Value
          averageValue: 3600k

However, the HPA errors out as shown below:

➜  ~ kubectl describe horizontalpodautoscaler coolgarden-hpa
Name:                               coolgarden-hpa
Namespace:                          coolgardener
Labels:                             cattle.io/creator=norman
Annotations:                        field.cattle.io/creatorId: u-x5rfkgkxtg
                                    field.cattle.io/displayName: mc-hpa-1
CreationTimestamp:                  Tue, 13 Apr 2021 12:59:15 -0700
Reference:                          Deployment/coolgardener
Metrics:                            ( current / target )
  "coolgarden_request_age" on pods:  <unknown> / 3600k
Min replicas:                       1
Max replicas:                       3
Deployment pods:                    2 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedComputeMetricsReplicas  57m (x13 over 60m)   horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
  Warning  FailedGetPodsMetric           32s (x234 over 60m)  horizontal-pod-autoscaler  unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)

The pods.custom.metrics.k8s.io * address seems to be incorrect (based on kubectl --raw output above. Not sure where this is being set and how this can be updated.

Anything I’m doing wrong here?

2

The issue here turned out to be that this cluster had the following apiservices installed:

kubectl get apiservices | grep -i custom 
v1beta1.custom.metrics.k8s.io           cattle-monitoring-system/rancher-monitoring-prometheus-adapter   True                      80m
v1beta1.external.metrics.k8s.io         custom-metrics/custom-metrics-apiserver                          False (ServiceNotFound)   20d
v1beta2.custom.metrics.k8s.io           custom-metrics/custom-metrics-apiserver                          False (ServiceNotFound)   20d

I removed v1beta1.external.metrics.k8s.io and custom metrics started working correctly.