Adding monitoring rules with Prometheus operator

Rancher
1

Hi!

I am trying to add additional monitoring rules to Rancher’s Prometheus programmatically by deploying a Kube manifies of kind PrometheusRule.

Basically, it should work like adding additional queries to store into new metrics https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules

For example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app: prometheus
  name: promtest
  namespace: cattle-prometheus
spec:
  groups:
  - name: test-group
    rules:
    - expr: |
        sum by (instance) (apiserver_request_latencies_count)
      record: instance:latency

The problem is, that no such new metric instance:latency is recorded.

I also tried the same with a separate Prometheus deployment (and Rancher-Monitoring turned off) from https://github.com/coreos/kube-prometheus and it works correctly.

Has anyone successfully added additional Prometheus rules to the existing Cluster-based Prometheus monitoring?

Cheers,
Damian

2

hi, have you found a way? I have the same problem as you.

3

Hi! Yes indeed!
Since a while we started provisioning and configuring Rancher with Hashicorp Terraform which is a bless!
Setting alert rules can be done with this resource e.g. https://www.terraform.io/docs/providers/rancher2/r/clusterAlertRule.html

I can really advise to use terraform for automation of Rancher!

Thanks to the authors of the provider!

Best regards, Damian

More links: https://rancher.com/blog/2019/rancher-2-terraform-provider/
https://youtu.be/Vc0EA018FrU