Adding alert configuration to Prometheus Service and Alertmanager Service programmatically (Dynamic Configuration of Alerts)


#1

How to dynamically add alerts rules in rules.conf and Prometheus yml file via API or something in Prometheus?

I have added Prometheus stack for monitoring Rancher server. I have tried configuring alerts in Prometheus stack by manually, it worked.

Now I want to add the rules to prom-conf and configure web hooks in alert manager config yml file dynamically. Is there anyway I can do that?? Do they have API for this scenario?


#2

have you found a way ?


#3

Yes. I have used ansible for this purpose. Downloaded machine config file when host becomes active and add it to the ansible inventory file. I have used ansible playbook file to connect to the prometheus stack deployed host with help the ansible inventory (passwordless ssh connection to that specific host) and added the alert rules in prometheus yml file. I just need run ansible file for adding alert rules.