host1 is in the cloud, host2 and host3 are on premise (home network) behind wi-fi router/firewall
Rancher server is deployed on host1.
The problem: It seems like only one of host2 or host3 can be part of the Rancher IPSec network at a time. Host1 has no such problem, so it’s only for hosts running onprem.
Symptom: rancher healthcheck container, running on the host that is having problems, is stuck in the ‘Initializing’ state. Cross host/container ping doesn’t work for containers deployed on this host.
I can ‘fix’ host that is in this state (let’s say host2) by rebooting it. Host reboot will generate following sequence (i can observer it in the /env/1a5/infra/hosts page):
host2 status Active -> Reconnecting->Active
host2 healthcheck status Initializing->Stopped->Initializing->Running(green)
host3 healthcheck status Running->Stopped->Initializing
host3 becomes non operational and host2 is ‘fixed’ i.e. i can ping it’s containers from host1
This looks to be a limitation in your home router. Rancher uses IPSec to send traffic between hosts, and a lot of home routers do not have good support for multiple IPSec connections to the same destination.
Each host needs to be able to reach every other host using the registered IP addresses. If there’s one or more external hosts, they will not be able to reach multiple in-house hosts separately.
This is never going to work with more than one host inside a typical home situation where you have one public IP, unless you add an additional VPN layer or similar on top to give them all different register-able addessses.
It means nobody behind NAT can add more than one host to the IPSec overlay without creating VPN (as a side effect it’s gona be secure as hell I suspected something like this but was hoping for some ‘magic’ that can be done by router.
I’m still going to try and use one host in home network as a proxy running a lot of load balancers connected to the internal ‘external services’ running on other local hosts. Will have to figure out how to manage containers running on them, may be even deploy another Rancher server that is for local network only…
I solved this by setting up separate environments and creating ‘external services’ from the cloud through to the home and internal services which reside internally.
Just to make sure I understand your setup, couple of questions:
Did you managed to get ‘Internal Hosts.(host 2/3)’ connected to the Rancher Server without problems?
Is your Rancehr Server running in the cloud somewhere or locally?
I knew the way I explained it might not have been too intuitive.
Start with a local VM running RancherOS/Server and connect local VMs as hosts to it. I’m using keepalived and a ssl loadbalancer to create a proxy for services ran within this environment. service.domain.local ->10.1.1.254:443 for example
Then spin up an AWS instance and connect it to essentially 8080 at home. Then use a VM internally (on a dmz net) to connect to the internal ip of the rancher server:8080 and used CATTLE_IP= [public ip]. This means that the host on AWS and internally can connect via IPSEC.
Then it gets a little trickier. Your “external” environment has a bastion host that can connect to service.domain.local for any services in that environment. And use an “External Service Alias” to your internal services running on your internal host.
When a service in your external environment (like another lb or nginx etc) and calls to that service it’ll use the loadbalancer and will pass that data over IPSec between hosts.
At least it works for me. I’ll try to put together a formal write up in the next few days.
Chris, I’m glad to hear it works for you, will keep your solution as a back up plan if I hit the wall with whatewer I’m trying to do. Having write up will help a lot for sure.
The main difference between your set up and what I’m trying to do is the location of Rancher Server, in my case it is running in the Cloud and I want to keep it there. In this way I can relly on it even if my home internet/network is down.
I figured if i just deactivate the host(s) in local network I still can start/stop containers using Ranchers and it provides some limited visibility of what is running on the host (i can even see the container logs using Rancher). So, I have only one host that is getting all the IPSec traffic and this host is running Nginx proxy that is ‘visible’ by the LB running in the cloud. All other ‘home network’ hosts have the network/methadata/ipsec Rancher services disabled.
Just a question here … If we want to do a hybrid or register a pubic Cloud cluster into an on-premise Rancher 2.0 master our data center will need to have a public IP, correct ?
Isn’t that a security Risk? The Cloud image get’s compromised and having a
vpn link back to the datacenter would allow the attacker to gain access to
on-premise resources, unless the master is then in a DMZ area.
I suspected something like this but was hoping for some ‘magic’ that can be done by router.