DNS issues with Rancher 1.6.14

Rancher 1.x
#1

Dear all,

I’m completely new to Rancher, so please bear with me :wink:

I have some questions regarding DNS issues on Rancher 1.6.14:

At work we use a Rancher environment for CI. This environment has many stacks, one of them based on Grafana (Docker Hub). In this stack we have 2 Grafana related containers, grafana (7.3.4) and grafana-db (running PostgreSQL 9.6.5).

In order to get things running, we had to set the GF_DATABASE_URL variable to postgres://grafana:grafana@some.ip:5432/grafana. “some.ip” is obviously the IP of the grafana-db container. I understand this is bad practice in an environment using floating IPs, but when we tried to use grafana-db instead of the IP, we got an error because grafana-db could not be resolved…

I followed the procedure documented at Troubleshooting FAQs about Rancher, but to little avail.

What am I doing wrong ?

Best regards,

Samuel

#2

If you are completely new to Rancher, please don’t use 1.6 anymore (rancher/server image) but use 2.x which was introduced in 2018 and currently at v2.5.5 as latest release (Release Release v2.5.5 · rancher/rancher · GitHub)

If you need to use 1.6 (please don’t), you can check DNS Service in Rancher to see how DNS is supposed to work. Based on that, please indicate how you are launching stacks/services and if all the services in the infrastructure stacks are running (mostly metadata and dns)

#3

@superseb : thank you very much for your quick reply :slight_smile:

Based on the documentation you sent me, I checked the rancher-compose.yml file associated with this stack, but couldn’t find any mention of “links”. Regarding the infrastructure stack, there are 6 metadata containers running, and 6 dns containers running, albeit without an IP address (I don’t know whether that’s relevant or not).

I also followed this procedure: Troubleshooting FAQs about Rancher

root@7cee91b78b6b:/# cat /etc/rancher-dns/answers.json | grep 192.168.40.7

grafana-db.grafana-private.rancher.internal.":{"answer":["192.168.40.7"]},"grafana-private-grafana-1.rancher.internal.":{"answer":["192.168.40.8"]},"grafana-private-grafana-db-1.rancher.internal.":{"answer":["192.168.40.7"]},

But:

bash-5.0$ ping grafana-db
ping: bad address 'grafana-db'

What am I doing wrong ?

Best regards,

Samuel

#4

Please share the resource files you are using to create the stack. The linked explanation on DNS service states it automatically resolves in a stack without links. Are the services in the same stack? Does it work with service.stack or service.stack.rancher.internal?

There are more examples in Internal DNS Service in Cattle Environments

#5

Hi @superseb :slight_smile:

I need to check with my colleagues whether I can share the resource files online or not. The services grafana and grafana-db are indeed in the same stack (grafana-private). From a shell on grafana:

bash-5.0$ ping grafana-db.grafana-private
ping: bad address 'grafana-db.grafana-private'

bash-5.0$ ping grafana-db.grafana-private.rancher.internal
ping: bad address 'grafana-db.grafana-private.rancher.internal'

Best regards, Samuel