Howdy!
New to Rancher here, and still figuring things out. I am deploying AWX in rancher, using postgresql operator for the postgresql requirements, and trying to enable metrics collection and monitoring via the built in Rancher Prometheus and Grafana instances.
I was able to configure custom metrics for postgresql and I can view them in the grafana dashboard provided by Rancher. Everything working great there. Now, I am trying to collect metrics from AWX, and have found that I need to log into AWX before I can view metrics (whereas postgresql required no auth). I can log into the AWX site at https:// my.awx.instance and then manually browse to the endpoint “https:// my.awx.instance/api/v2/metrics” and see all my metrics. I can also get there via the node private IP http:// privateIP:30812/api/v2/metrics/ . I have configured the custom metrics for my app/pod in Rancher (I’ve tried both 80 and 30812, which is what the AWX nodeport svc is using for ports, and /api/v2/metrics/ for the path), and I see my endpoint in the Rancher-provided Prometheus page when I look at “status > targets”… however… it shows the following error next to my endpoint:
Get http:// 192.168.40.212:80/api/v2/metrics/: dial tcp 192.168.40.212:80: connect: connection refused
I assume this is because Prometheus needs to authenticate to access my metrics. I found in the AWX documentation that you can monitor AWX with prometheus by supplying a custom configuration for prometheus.yaml that includes a bearer_token entry (see here: https://docs.ansible.com/ansible-tower/latest/html/administration/metrics.html) . I have created a metrics account and bearer token, but cannot figure out where in Rancher to supply my token and instruct prometheus to use it.
Can anyone provide pointers? Hours of google searching has not turned up any advice. Thanks in advance!