Collecting HAProxy stats - multiple LB container for the service

Rancher 1.x
1

The scenario is this:

  • Load Balancer service running with “Always run one instance of this container on every host”.
  • HAProxy config set to expose stats.
    listen stats
    bind 0.0.0.0:9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /haproxy_stats
    stats auth Username:Password
  • I need to collect the HAProxy stats for each servers.
  • I know how many instance and I know their external IP address (host IP). That’s a fix set so I can have the collecting software set up with something like: servers: [10.10.10.1:9000, 10.10.10.2:9000].

Ideas I had and why it did not work:

  • Make HAProxy containers expose port 9000 - in the UI at least, there isn’t really a way to do that. If I add a port, it tries to forward that to a backend.
  • Run the collecting software as a service in Rancher with a link to the LB service. Works except it collects from one of the two instances of the LB through DNS round-robin. I don’t have a way to specify which instance (or both). I could run two collectors containers - one per host like the LB - so long as I can guarantee that they collect the stats from the LB container running on the same host.
  • If I could add a sidekick to the LB service, which would be the collector, I could collect the stats through a socket instead of IP.

Any ideas? Something I missed and didn’t think about on how to do this?

2

+1

Ran into same issue. This should be an option on LB deploy (Expose Port + Enable Stats Page). Would be nice to have that page to drop some containers into MAINT mode.

3

Can you file an issue in Github?

4
5

@olds463 - I got a solution now, if you are interested.

Here is how I am monitoring my load balancer on Rancher:

The load balancer service is configured with the following in the global configuration section:

listen stats 
bind 0.0.0.0:9000
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /haproxy_stats

My performance monitoring is done using InfluxDB as database and Telegraf to collect data, so I configured a Telegraf service in the same environment with the following input:

[[inputs.haproxy]]
  servers = ["http://mystack_lb_1.lb.mystack:9000/haproxy_stats;csv", "http://mystack_lb_2.lb.mystack:9000/haproxy_stats;csv"]

As covered in the issue linked earlier, in Rancher v0.63.0, you can address the container instance through the internal DNS as: <instance-name>.<service-name>.<stack-name>

In my case, I know I have exactly 2 containers running, so I can statically configure Telegraf to pull those. In a more complex scenario where LB instance come and go, I guess one could write some code to dynamically change the configuration.

6

http://etlweather.github.io/2016-03-20-rancher-haproxy-monitoring/

7

I’ve got a solution to expose the port of the internal load balancer, just in case anyone needs this:

Basically I have created a service alias in my stack which points to my load balancer, then i have configured my load balancer to route requests on port 9000 to the service alias on port 9000. This way I can access the ha proxy stats without using an additional container.