Unable to fetch metrics from custom metrics API in Rancher v2.5

I am trying to set up horizontal pod autoscaling with custom metrics in Rancher v2.5. I installed rancher-monitoring following instructions here. This installed prometheus-adapter in the cluster as well.

I updated the prometheus-adapter configMap to add my custom metric as shown below (truncated output for brevity):

➜  ~ kubectl -n cattle-monitoring-system get configmap rancher-monitoring-prometheus-adapter -o yaml 
...
      - seriesQuery: 'CoolGardenRequestListener_requests_age_milliseconds_max{namespace!="",pod!=""}'
        resources:
          template: <<.Resource>>
        metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
        name:
          matches: ^(.*)
          as: "coolgarden_request_age"
...

I ran the following to ensure this metric was available via k8s api:

➜  ~ kubectl get --raw "/k8s/clusters/c-t66dx/apis/custom.metrics.k8s.io/v1beta1/namespaces/coolgardener/pods/*/coolgarden_request_age" | jq 
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/coolgardener/pods/%2A/coolgarden_request_age"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "coolgardener",
        "name": "coolgardener-6f9cf4595-klrz9",
        "apiVersion": "/v1"
      },
      "metricName": "coolgarden_request_age",
      "timestamp": "2021-04-13T20:52:52Z",
      "value": "19373868",
      "selector": null
    },
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "coolgardener",
        "name": "coolgardener-6f9cf4595-v4xrt",
        "apiVersion": "/v1"
      },
      "metricName": "coolgarden_request_age",
      "timestamp": "2021-04-13T20:52:52Z",
      "value": "78170499",
      "selector": null
    }
  ]
}

Then I proceeded to create an HPA using the following:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: coolgarden-hpa
  namespace: coolgardener
spec:
  maxReplicas: 3
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coolgardener
  metrics:
    - type: Pods
      pods:
        metric:
          name: coolgarden_request_age
        target:
          type: Value
          averageValue: 3600k

However, the HPA errors out as shown below:

➜  ~ kubectl describe horizontalpodautoscaler coolgarden-hpa
Name:                               coolgarden-hpa
Namespace:                          coolgardener
Labels:                             cattle.io/creator=norman
Annotations:                        field.cattle.io/creatorId: u-x5rfkgkxtg
                                    field.cattle.io/displayName: mc-hpa-1
CreationTimestamp:                  Tue, 13 Apr 2021 12:59:15 -0700
Reference:                          Deployment/coolgardener
Metrics:                            ( current / target )
  "coolgarden_request_age" on pods:  <unknown> / 3600k
Min replicas:                       1
Max replicas:                       3
Deployment pods:                    2 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedComputeMetricsReplicas  57m (x13 over 60m)   horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)
  Warning  FailedGetPodsMetric           32s (x234 over 60m)  horizontal-pod-autoscaler  unable to get metric coolgarden_request_age: unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get pods.custom.metrics.k8s.io *)

The pods.custom.metrics.k8s.io * address seems to be incorrect (based on kubectl --raw output above. Not sure where this is being set and how this can be updated.

Anything I’m doing wrong here?

The issue here turned out to be that this cluster had the following apiservices installed:

kubectl get apiservices | grep -i custom 
v1beta1.custom.metrics.k8s.io           cattle-monitoring-system/rancher-monitoring-prometheus-adapter   True                      80m
v1beta1.external.metrics.k8s.io         custom-metrics/custom-metrics-apiserver                          False (ServiceNotFound)   20d
v1beta2.custom.metrics.k8s.io           custom-metrics/custom-metrics-apiserver                          False (ServiceNotFound)   20d

I removed v1beta1.external.metrics.k8s.io and custom metrics started working correctly.