How to connect home network and external worker node

Rancher 2.x
1

Hi! I asked this question already on github, but maybe you can help me with that here.
Here is my question:

Hi. I`m using rancher cluster inside “home” network with nat. My local network in 192.168.1.0/24 range and i have 3 worker nodes with 1.135, 1.136, 1.137 ips.

I use NAT in my mikrotik router to get internet connection. I want to setup new worker node in AWS cloud, so i used new node with NAT in cloud.

I connected this node via l2tp and got ppp0 interface:

4: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UNKNOWN group default qlen 3
    link/ppp 
    inet 192.168.1.126 peer 192.168.1.1/32 scope global ppp0
       valid_lft forever preferred_lft forever

Also i setup custom routes to get my home local network:

192.168.1.0/24 dev ppp0 scope link 
192.168.1.1 dev ppp0 proto kernel scope link src 192.168.1.126

I want to setup worker node inside rancher cluster, so here is my rancher-cluster.yml (rke) config for this node:

  - address: rkework.yac.loc.s-hub.xyz
    internal_address: 192.168.1.126
    user: ila
    role: [worker]
    ssh_key_path: /home/ila/.ssh/id_rsa

So the problem is that, as i understand, rancher network (canal by default) using first interface, e.g. eth0 for interconnect and flannel using eth0 also. Thats why i can’t ping/curl/do anything else from new node in “clouds” because flannel using eth0 instead of ppp0.

Is there any way to configure custom IFACE (or canal_iface) only just for one node? If yes, how. If no, what to do then?

I hope you can help me with this problem.

I also tried to remove old routes

10.128.0.0/24 dev eth0 proto kernel scope link src 10.128.0.35 
10.128.0.1 dev eth0 proto dhcp scope link src 10.128.0.35 metric 100

and create this one

10.128.0.0/24 dev ppp0 proto kernel scope link src 10.128.0.35 
10.128.0.1 dev ppp0 proto dhcp scope link src 10.128.0.35 metric 100

But this does not help also

2

I’m not sure how easy/feasible this will be and with you being behind a NAT that’s not great either, but it could be possible? So canal is calico + flannel, and calico uses BGP which is the same thing Routers on the Internet use to talk and you can technically set up calico to talk BGP with things outside your Kubernetes cluster to allow connections to your internal pod network (see horrifying example at Run the HAProxy Kubernetes Ingress Controller Outside of Your Kubernetes Cluster - HAProxy Technologies for someone doing it). I also heard somewhere that the cloud providers use Calico.

So I think it might be theoretically possible, I suspect it’ll be painful, and those are the breadcrumbs I’m aware of to work with (i.e. you might search Calico too and not just canal).

I wish you good luck.