Custom metrics in project monitoring

Rancher
1

I don’t understand what is the purpose of project monitoring. It seems that me, that project-level prometheus is only configured to scrape configs from its own namespace. How can I add my own applications? Do I need to run prometheus on my own if I want to monitor my applications?

2

I have successfully added, e.g., kube-state-metrics metrics to the cluster-wide Prometheus monitoring. It works by adding the workload’s service annotations the entry prometheus.io/scrape=true

Adding project-based Prometheus should enable you to display only metrics from the namespaces within that project. Again, to add your app’s /metrics to scraping you can use the annotation above.

3

@damlub Well, as far as I can see, that only works in the cluster-wide Prometheus instance and it’s limited to the system namespaces. I can see this in the cluster-wide Prometheus config:

  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - ingress-nginx
      - ingress-controller
      - kube-system

Project-level Prometheus doesn’t have this the “prometheus-io-scrape” job.

I have deployed the redis-ha helm chart and it includes these annotations:

|prometheus.io/path|/metrics|
|prometheus.io/port|9121|
|prometheus.io/scrape|true|

I can’t find it in Prometheus, so I wanted to confirm if the Rancher-managed instances are only meant to Rancher services.

4

The Rancher documentation does not make this clear, but the metrics collection is configured using the prometheus-operation (https://github.com/coreos/prometheus-operator). This means you have to create ServiceMonitor objects to setup scraping.

I have successfully scraped custom metrics from my application by enabling cluster-level and project-level monitoring. Then creating a ServiceMonitor similar to this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
spec:
  endpoints:
  - path: /metrics
    port: metrics9992
    scheme: HTTP
    targetPort: 9992
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app: myapp
5

Just FYI - Rancher can add the ServiceMonitor objects for you, when editing a workload, under ‘Show advanced options’ and ‘Custom Metrics Endpoints’

https://rancher.com/docs/rancher/v2.x/en/project-admin/tools/monitoring/#project-metrics

6

@derfred Thanks for clearing that up, the project-monitoring documentation is not really clear on this.

Is the project-monitoring Prometheus resource managed by the cluster-monitoring PrometheusOperator?
My ServiceMonitor in namespace cattle-prometheus-p-xy is not picked up and rendered into the project-monitoring Prometheus, although the serviceMonitorNamespaceSelector and serviceMonitorSelectormatch on labels.

How did you manage to get it working?

7

@derek I tried creating my own custom metrics endpoints, but neither do I see any ServiceMonitor being created, nor my custom metrics endpoint being populated into any of the Prometheus scrape configs…

How did you get it working with “Custom Metrics Endpoints”? Thank you in advance.

8

@derfred @derek: Nevermind… my cluster had a completely unrelated issue and did not update the label field.cattle.io/projectId on new namespaces. :see_no_evil:
After fixing the issue, both your options worked as expected.