Alerting is dead, missing alertmanager-operated

Rancher
1

Seems we are missing some bits, and alerting is not working. I see this in logs:

2020/02/27 16:56:20 [ERROR] ClusterAlertGroupController c-ctt27/event-alert [cluster-alert-group-controller] failed with : Failed to get service for alertmanager, service “cattle-prometheus/alertmanager-operated” not found

Indeed there is no “alertmanager-operated” in this cluster, and I don;t see any workload this would have likely pointed at.

How can I add it back in?

2

I was able to fix this issue by removing the notifier for each alert group, saving, then re-adding. This caused the clsuter to redeploy the missing service and deployment