Alertmanager configs in Monitoring V2 in Rancher no working

Rancher V2.7.5 and v2.7.6 both have the same issue, notifications via Slack or Email Receivers are not sent
Followed instruction here How to set up Alertmanager configs in Monitoring V2 in Rancher | Support | SUSE

There are also opened issues here

I tried with Robusta in an AKS env and just worked, their alert manager sent notifications to Slack or Emails based on Prometheus

At the moment we are not receiving alerts from production Rancher clusters or even testing ones, different clouds are all behaving the same

Would be great if someone could have a look into this

Hi, in the Alertmanager GUI → Status section, can you see your alert manager config present there? We need to check if the Config is getting reflected in the Alertmanager. If the Alertmanager configurations is not getting updated then search for alertmanager-monitoring-rancher-monitor-alertmanger secret in the cattle-monitoring-system namespace. You need to delete this secret since it contains the alertmanager data and after deleting this, new config will get applied. Wait for some time and if the config is correct it will be updated.

Hi @vaishnav,
Tried your steps, easy to follow, and indeed was the wrong config, however after deleting the secret still showed the wrong config, should be a slack and email receivers instead of pagerduty, which we don’t use
In addition the Status appears disabled, I have tried to read few internet article and Suse documentation but this is not really clear about how to proceed for enabling it

This is the confusing part
Routers and Receivers appear as deprecated, receivers are empty but routes have a default route that matches the above, this “default null” looks cannot be deleted

My configuration is set in AlertmanagerConfigs section but looks not been applied

If it’s showing disabled it’s okay. And can you create a AlertManagerConfig instead of Routes and Receivers? You can configure the routes and receivers in the AlertManagerConfig as well.