Custom metrics in project monitoring

I don’t understand what is the purpose of project monitoring. It seems that me, that project-level prometheus is only configured to scrape configs from its own namespace. How can I add my own applications? Do I need to run prometheus on my own if I want to monitor my applications?

I have successfully added, e.g., kube-state-metrics metrics to the cluster-wide Prometheus monitoring. It works by adding the workload’s service annotations the entry

Adding project-based Prometheus should enable you to display only metrics from the namespaces within that project. Again, to add your app’s /metrics to scraping you can use the annotation above.

@damlub Well, as far as I can see, that only works in the cluster-wide Prometheus instance and it’s limited to the system namespaces. I can see this in the cluster-wide Prometheus config:

  - role: pod
      - ingress-nginx
      - ingress-controller
      - kube-system

Project-level Prometheus doesn’t have this the “prometheus-io-scrape” job.

I have deployed the redis-ha helm chart and it includes these annotations:


I can’t find it in Prometheus, so I wanted to confirm if the Rancher-managed instances are only meant to Rancher services.

The Rancher documentation does not make this clear, but the metrics collection is configured using the prometheus-operation ( This means you have to create ServiceMonitor objects to setup scraping.

I have successfully scraped custom metrics from my application by enabling cluster-level and project-level monitoring. Then creating a ServiceMonitor similar to this:

kind: ServiceMonitor
  name: myapp
  - path: /metrics
    port: metrics9992
    scheme: HTTP
    targetPort: 9992
      insecureSkipVerify: true
      app: myapp

Just FYI - Rancher can add the ServiceMonitor objects for you, when editing a workload, under ‘Show advanced options’ and ‘Custom Metrics Endpoints’