K8s endpoint authorization

Rancher 2.x
1

Hello.

I attempt to setup cluster monitoring with Prometheus. I want it to be an external to the cluster, so i going not to use build-in prometheus deployment of Rancher v2.2.2.

I have created Bearer token for user in Rancher, and using it i succesfully scraped metrics about nodes and points. But i cannot get metrics of endpoints.

image
image1610×588 45 KB

Here is part of my prometheus.yml:

# =========================================
# metrics for default/kubernetes api's from the kubernetes master
# =========================================
- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://192.168.100.15/k8s/clusters/c-lhfkr
    tls_config:
      insecure_skip_verify: true
    bearer_token: "token-8mtmm:qbnzv2z4rxwwbtmns787lqrt6f6qqlnw5kghv9qppvht4jzz27fqvb"
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token: "token-8mtmm:qbnzv2z4rxwwbtmns787lqrt6f6qqlnw5kghv9qppvht4jzz27fqvb"
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

So it succesfully scraped service discovery info, but then Prometheus try to scrape metrics of kube-apiserver failed with 401 Unauthorized.

I think, it is because request to “https://192.168.100.11:6443/metrics” bypassing Rancher auth proxy. I can manually create service account and generate token for it in k8s, and it works. But i want to use token from Rancher…

How can i solve it?

2

Okay.

There is a http://rancher/k8s/cluster/c-lhfkr/metrics link. What does this metrics for? I believe, it is for API endpoint, but which one?