Access container by hostname on internal Rancher network

Rancher 1.x
1

I’m having lots of fun playing around with Rancher so first of all thanks for building such an amazing product. I’ve been thoroughly impressed with the system and everything that supports it.

Following an old Rancher blog post I’m running an OpenVPN server (rancher-vpn-server) in my environment. I am then able to successfully VPN right in to the Rancher network and access containers by their IP address.

However, I’m struggling to understand or find the documentation related to the internal DNS system and I’ve not been able to access containers by a hostname or DNS entry.

How does this aspect work and how might I configure it so that I can VPN in and access a container by a DNS name such as redis.my.rancher.com (or even redis.rancher.internal which I can’t get working either)?

2

Would the DNS changes in 0.46 help with this?

3

Have you read about the container hostname override? Currently, we use a identifier as the hostname, but it can be overridden in the container with the container_name.

http://docs.rancher.com/rancher/labels/
Search for l io.rancher.container.hostname_override: container_name

4

Hi denise thanks for getting back to me.

I have upgraded to 0.46 and tried using the label you suggest to no avail.

If I used the label on a service I can execute shell on a container within that service and successfully curl to that hostname. For example: curl http://sonarqube_sonarqube_1:9000

However if I execute shell into any other container I get could not resolve host

And of course if I VPN into the rancher network I cannot access the container by this hostname.

5

The io.rancher.container.hostname_override: container_name setting gives the container a hostname I can work with but I am still unable to resolve this name from any other container. What am I missing? I thought from the 0.46 release notes that all names would be resolvable from all containers even if there were no explicit links.

6

What should resolv.conf look like on any given container? I’m convinced there is something wrong because I cannot get name resolution working no matter what I try. Are there some diagnostics I can perform from one of the network agents to try and isolate the issue?

7

@djskinner

I can work with but I am still unable to resolve this name from any other container

If that other container belongs to the same stack that the container you are trying to resolve belongs to? Containers are resolvable to other containers only within the same stack

What should resolv.conf look like on any given container

It should look like this:

root@040fdc90e1ce:/# cat /etc/resolv.conf
# nameserver 8.8.8.8
nameserver 169.254.169.250

where name server ip is a link local ip address of network agent

Are there some diagnostics I can perform from one of the network agents to try and isolate the issue

Given the use case where you try to resolve container C2 from container C1, you can check the contents of /var/lib/cattle/etc/cattle/dns/answers.json from the network agent running on container’s C1 host. Look up the section starting with the ip address of your container C1 - it would have all the record that are resolvable to C1. Check if you see C2 there. Example:

"10.42.173.28": {
        ....
        "a": {
        ....
        "C2.": {"answer": ["10.42.163.35"]},
        "C2.rancher.internal.": {"answer": ["10.42.163.35"]},
        ...........
        }
8

If that other container belongs to the same stack that the container you are trying to resolve belongs to? Containers are resolvable to other containers only within the same stack

I see. I can confirm this part works. curling any container from another container within the stack by container name does work as expected.

I think I may have misunderstood the changes brought in by 0.46 to mean that all containers would be discoverable with all other containers.

Is there to specify that a container is resolvable from anywhere else within the managed network?

Thanks for providing the further insights into resolv.conf and answers.json. Is the Rancher DNS server always at 169.254.169.250?

9

Is the Rancher DNS server always at 169.254.169.250

yes

Is there to specify that a container is resolvable from anywhere else within the managed network?
I think I may have misunderstood the changes brought in by 0.46 to mean that all containers would be discoverable with all other containers.

In the near future, we are planning to make event services cross stacks to be discoverable to each other (by serviceName.stackName). Today to make container resolvable cross stacks, you have to create link between services of different stacks.

10

@djskinner here is the ticket to keep track of “Discover services cross stacks by default” feature: