Having an odd problem on Rancher 0.35.0-rc1. Currently have a single VM in my install controlled from the VM’s host, which is where the Rancher control server runs. I use Cloudflare to provide SSL termination and other services for most of my websites, and all of these work fine. I also have 2 separate Rancher environments, each with a single VM host, and these can communicate. However, if I’m on one of the VMs, and I run curl against the load balancer on the same VM, the connection is refused.
My use case is a Docker registry running at docker.my.domain. This registry is hosted in a Rancher stack containing the registry itself and, eventually, a front-end Nginx for authentication. The registry is fronted by an SSL-terminated, Rancher-managed load balancer. Because Cloudflare’s free/pro plans limit the body size of uploads, I’ve had to disable Cloudflare on docker.my.domain.
I can push images to docker.my.domain off-site, and HTTP connections to it work fine. I can access docker.my.domain from other VMs. I cannot access docker.my.domain from the VM where it is hosted. The connection is simply refused. As such, any images needing to pull from my own hosted Docker registry fail because the subdomain resolves to the VM’s own IP.
This issue isn’t specific to Docker registries, that’s just the use case that caused me to hit this. I can’t access the load balancer IP at all from the VM because the connection is refused, regardless of what happens to be at the other end.
I’m using UFW as my firewall, but disabling that didn’t change this behavior. What am I missing?
I have another container, a simple HTTP-to-HTTPS redirecter, not hosted behind the load balancer but open to the public internet. This, too, works fine when I access it remotely, but fails when I try hitting my host’s IP from itself. So this doesn’t appear to be an issue with the load balancer specifically. The container is set to redirect public port 80 to its own port 80 so is a fairly simple setup.
Thoughts? Since this infrastructure is being accessed pre-launch, I can’t exactly link a container to the registry of its image. But I’d have expected to be able to access my own local infrastructure as if it were remote. I’ve done this before with Docker-based infrastructure hosted off of Docker-based load balancers, so it’s something with Rancher’s overlay networking perhaps?
You’re not missing anything, hairpin NAT (talking from a container on the host to the public IP of the host and back to a container on the host) just does not currently work with managed networking. #1920
On a side note, we also recommend using versions of rancher/server that do not have the “rc” in the name. Anything with a “rc” in the tag name means that it’s going through QA testing. If it doesn’t pass, then we cut another rc until QA gives a stamp of approval. At that time, we tag it as a version with no “rc” in the title.
Got it, thanks, thought I was going crazy for a bit. I’ll use the hacky solution of placing the Docker registry in a different environment on a different VM, since it should only be accessed from one internal location anyway. Everything else should be fine since it routes through Cloudflare.
Thanks for the recommendation re: RCs. Normally I’d just stick to stable, but the recently-resolved SSL certificate verification issues were going to make or break whether I used Rancher, and my setup simply would not have worked on anything before the recent RC. I’ll certainly stick to stable going forward.
Though in all fairness, I typically associate RC with “this will become the final release if no issues are found with it, so test in production if you can risk it” and Beta with “we’re still doing internal testing on this and make no guarantees about its relation to an actual release. Keep it away from production.” Is that just me? I’m used to project owners asking their users to hammer on RCs and assumed the same would be true here.
It’s a one week cycle, so it’s sort of an optimistically abbreviated version of that… there’s not really a lot of time for beta vs rc. We make a 0.X-rcY release (usually on Wednesday) and QA starts looking at it. They also look at master during the week, and there’s also automated testing. If there’s nothing majorly wrong, that does become the actual 0.X release for the week and non-blocking bugs go into the next week. Frequently there’s a couple important fixes, and rc2 or rc3 becomes 0.X.
They’re public and it’s fine to use them if you know what you’re getting into, but @denise is just saying we wouldn’t suggest a typical user pick up RCs (especially rc1) like you wouldn’t suggest a typical person install the Mac OS 10.11 RC on their computer today.
I’ll use the hacky solution of placing the Docker registry in a different environment on a different VM, since it should only be accessed from one internal location anyway. Everything else should be fine since it routes through Cloudflare.