Custom metrics in project monitoring

I don’t understand what is the purpose of project monitoring. It seems that me, that project-level prometheus is only configured to scrape configs from its own namespace. How can I add my own applications? Do I need to run prometheus on my own if I want to monitor my applications?

I have successfully added, e.g., kube-state-metrics metrics to the cluster-wide Prometheus monitoring. It works by adding the workload’s service annotations the entry prometheus.io/scrape=true

Adding project-based Prometheus should enable you to display only metrics from the namespaces within that project. Again, to add your app’s /metrics to scraping you can use the annotation above.

@damlub Well, as far as I can see, that only works in the cluster-wide Prometheus instance and it’s limited to the system namespaces. I can see this in the cluster-wide Prometheus config:

  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - ingress-nginx
      - ingress-controller
      - kube-system

Project-level Prometheus doesn’t have this the “prometheus-io-scrape” job.

I have deployed the redis-ha helm chart and it includes these annotations:

|prometheus.io/path|/metrics|
|prometheus.io/port|9121|
|prometheus.io/scrape|true|

I can’t find it in Prometheus, so I wanted to confirm if the Rancher-managed instances are only meant to Rancher services.

The Rancher documentation does not make this clear, but the metrics collection is configured using the prometheus-operation (https://github.com/coreos/prometheus-operator). This means you have to create ServiceMonitor objects to setup scraping.

I have successfully scraped custom metrics from my application by enabling cluster-level and project-level monitoring. Then creating a ServiceMonitor similar to this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
spec:
  endpoints:
  - path: /metrics
    port: metrics9992
    scheme: HTTP
    targetPort: 9992
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app: myapp

Just FYI - Rancher can add the ServiceMonitor objects for you, when editing a workload, under ‘Show advanced options’ and ‘Custom Metrics Endpoints’

https://rancher.com/docs/rancher/v2.x/en/project-admin/tools/monitoring/#project-metrics