I don’t understand what is the purpose of project monitoring. It seems that me, that project-level prometheus is only configured to scrape configs from its own namespace. How can I add my own applications? Do I need to run prometheus on my own if I want to monitor my applications?
I have successfully added, e.g., kube-state-metrics metrics to the cluster-wide Prometheus monitoring. It works by adding the workload’s service annotations the entry
Adding project-based Prometheus should enable you to display only metrics from the namespaces within that project. Again, to add your app’s /metrics to scraping you can use the annotation above.
@damlub Well, as far as I can see, that only works in the cluster-wide Prometheus instance and it’s limited to the system namespaces. I can see this in the cluster-wide Prometheus config:
kubernetes_sd_configs: - role: pod namespaces: names: - ingress-nginx - ingress-controller - kube-system
Project-level Prometheus doesn’t have this the “prometheus-io-scrape” job.
I have deployed the redis-ha helm chart and it includes these annotations:
|prometheus.io/path|/metrics| |prometheus.io/port|9121| |prometheus.io/scrape|true|
I can’t find it in Prometheus, so I wanted to confirm if the Rancher-managed instances are only meant to Rancher services.
The Rancher documentation does not make this clear, but the metrics collection is configured using the prometheus-operation (https://github.com/coreos/prometheus-operator). This means you have to create ServiceMonitor objects to setup scraping.
I have successfully scraped custom metrics from my application by enabling cluster-level and project-level monitoring. Then creating a ServiceMonitor similar to this:
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: myapp spec: endpoints: - path: /metrics port: metrics9992 scheme: HTTP targetPort: 9992 tlsConfig: insecureSkipVerify: true selector: matchLabels: app: myapp
Just FYI - Rancher can add the ServiceMonitor objects for you, when editing a workload, under ‘Show advanced options’ and ‘Custom Metrics Endpoints’
@derfred Thanks for clearing that up, the project-monitoring documentation is not really clear on this.
Is the project-monitoring Prometheus resource managed by the cluster-monitoring PrometheusOperator?
My ServiceMonitor in namespace
cattle-prometheus-p-xy is not picked up and rendered into the project-monitoring Prometheus, although the
serviceMonitorSelectormatch on labels.
How did you manage to get it working?
@derek I tried creating my own custom metrics endpoints, but neither do I see any ServiceMonitor being created, nor my custom metrics endpoint being populated into any of the Prometheus scrape configs…
How did you get it working with “Custom Metrics Endpoints”? Thank you in advance.