Well, we finally got our issue with the F5 resolved so now I am able to access our Rancher HA cluster and everything appears to be working as it should. Hosts have been added (one agent & one server per host)… is this advisable? The documentation didn’t mention anything about how that should be distributed. Should I have 3 HA nodes and then 3 separate hosts for running the agents?
My primary question has to do with the internal load balancer. Now that we have everything behind the F5, I’d like to start pointing paths to services and it was my understanding that this was possible. I was able to do it with 1.1 before the major re-design, but it involved changing some code in the containers to add an nginx proxypass. Also, static content wasn’t displaying properly. Now with 1.3.1, I can’t get the load balancer to work at all and I wonder if there’s still some issue with headers not being forwarded properly on the F5.
Here is what I want in general:
ourdomain.com/app1 (443) -> app1 (80)
I have tried giving the container only a private port (80) and then assigning ourdomain.com as the request host with a port of 443. That is directed to /app1 and target is app1 on port 80. This is how I set it up in 1.1 and it mostly worked. Has it changed drastically and I’m just not understanding it? Right now I get a 404 no matter what I do. I haven’t been able to track down anything in any logs and I don’t see any traffic going to any of the hosts when I try to hit the ourdomain.com/app1 address. I’m a bit lost as to where the problem is, but I suspect it’s still the F5.
We have several web apps that are being developed for campus right now and will deployed in March. I’d like to demo this whole platform for several other departments on campus. I’m hoping that if they’re onboard with this, they’ll pick up the tab for enterprise support. We as a department just can’t foot that kind of bill.
Finally, is there any way to add an ACL of some kind to limit access to the Rancher admin site to only a certain range of IP’s? I don’t like the idea of exposing that to all of campus, even if it is limited via LDAP. Maybe I’m thinking about this all wrong. Any help would be greatly appreciated.
Thanks!
As a follow-up, a few other thoughts…
If I wanted to access a service behind the load balancer on a specific port, would that even be possible? The F5 is configured to forward all requests on 80 to 443 and then pass that to the hosts on 8080. Would an irule even be possible? I could always access the service on the host directly but then if that host dies and the service moves to a new host, it would no longer work.
In a scenario where there is one rancher server and multiple separate hosts in a non-HA configuration, I could control access to services via our external firewall. With the configuration we have now, that is not possible. Would I benefit more in this case from moving to a non-HA config with an external database and just launch different services on different hosts as needed? Just looking for some ideas here.
Thanks!
HA is mostly intended to be used on machines dedicated to hosting it. In general if the cost/maintenance of that is a concern, you probably don’t really need HA and would be better off with a simpler single server container pointed at a resilient database. In the event of the container failing all the hosts continue running their workloads and you can just start up another one pointed at the database.
The implementation of balancing has changed to be more configurable but the basic way it works is still the same. Where is the 404 coming from? (response headers may give a clue.) . The fact that it’s 404 and not 5xx suggests that something getting the request thinks it should be responding to it.
From what it sounds like you want you should have ourdomain.com A/CNAME a-dedicated-ip.on.your.f5, which does TCP (not HTTP, since you’re not decrypting the traffic here) forwarding to the IPs of the 3 hosts. Then a load balancer service deployed on those services with a rule for Public, HTTPS, ourdomaim.com, 443, /app1 -> app1, 80.
You can certainly for you to setup ACLs to restrict access but we feel this is mostly outside the scope of Rancher. Access controls performed inside the application are generally less secure than doing it at a lower more standard level like iptables (For example buffer overflows in the request handling that can be triggered even if the code will eventually decide this request should be blocked).
Thanks for the reply! Is there still a requirement that HA nodes be run in either 3 or 5 node configurations? I didn’t see any mention of this in the documentation since 1.2.
I’m fairly certain that what you’ve described for the F5 config is how we have is set up, though I can’t see the rules for the F5 so I’m not entirely sure. I’ll talk with the admin that controls that and get more information. As I understood it, it’s forwarding all 443 -> 8080 on the nodes. From there, I have the load balancer configured as you described. It’s just returning 404. I’ll see if I can get more info from the headers.
I really appreciate your help! I’ve been pushing this platform here for awhile. I’m a big fan. All of the devs in my department have bought into the new process and are completely on-board. My hope is that I can get the rest of campus on board with this if all goes well.
We test 3 & 5, though that seems to have been lost in the docs… More than 5 is overkill. 2 or 4 should work but we haven’t seriously tested it. The main concern would be what happens with 2 when one fails as you now have exactly half instead of a majority quorum.