I don’t understand what is the purpose of project monitoring. It seems that me, that project-level prometheus is only configured to scrape configs from its own namespace. How can I add my own applications? Do I need to run prometheus on my own if I want to monitor my applications?
I have successfully added, e.g., kube-state-metrics metrics to the cluster-wide Prometheus monitoring. It works by adding the workload’s service annotations the entry
Adding project-based Prometheus should enable you to display only metrics from the namespaces within that project. Again, to add your app’s /metrics to scraping you can use the annotation above.
@damlub Well, as far as I can see, that only works in the cluster-wide Prometheus instance and it’s limited to the system namespaces. I can see this in the cluster-wide Prometheus config:
kubernetes_sd_configs: - role: pod namespaces: names: - ingress-nginx - ingress-controller - kube-system
Project-level Prometheus doesn’t have this the “prometheus-io-scrape” job.
I have deployed the redis-ha helm chart and it includes these annotations:
|prometheus.io/path|/metrics| |prometheus.io/port|9121| |prometheus.io/scrape|true|
I can’t find it in Prometheus, so I wanted to confirm if the Rancher-managed instances are only meant to Rancher services.
The Rancher documentation does not make this clear, but the metrics collection is configured using the prometheus-operation (https://github.com/coreos/prometheus-operator). This means you have to create ServiceMonitor objects to setup scraping.
I have successfully scraped custom metrics from my application by enabling cluster-level and project-level monitoring. Then creating a ServiceMonitor similar to this:
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: myapp spec: endpoints: - path: /metrics port: metrics9992 scheme: HTTP targetPort: 9992 tlsConfig: insecureSkipVerify: true selector: matchLabels: app: myapp
Just FYI - Rancher can add the ServiceMonitor objects for you, when editing a workload, under ‘Show advanced options’ and ‘Custom Metrics Endpoints’