Issue With DNS on v0.63.1

Archive Rancher 1.x
1

Hi,

I’ve got a DNS issue since upgraded in version v0.63.1

I’ve got a container linked to a db container and I can’t resolv it using my mysql client neither using curl while the resolution seams ok on a dig or nslookup point of view.

[root@cloudsso-northapi:/]#curl cloudsso-db-master
curl: (6) Couldn't resolve host 'cloudsso-db-master'
[root@cloudsso-northapi:/]#curl "http://cloudsso-db-master"
curl: (6) Couldn't resolve host 'cloudsso-db-master'

but
[root@cloudsso-northapi:/]#nslookup cloudsso-db-master
Server: 169.254.169.250
Address: 169.254.169.250#53

Name:   cloudsso-db-master
Address: 10.42.217.120

==>Working using directly the IP
[root@cloudsso-northapi:/]#curl 10.42.217.120:3306
5.7.11-logT=]s`Op��S��|fc XPB l)mysql_native_password�Got packets out of order

==>Not working using the Name…
[root@cloudsso-northapi:/]#curl cloudsso-db-master:3306
curl: (6) Couldn’t resolve host ‘cloudsso-db-master’

Sebastien

2

Hi,

an important point not said, is that cloudsso-db-master is searhed from stack A to stack B

so it is an external_links which is used in docker-compose.yml.

I just tried to put them in the same stack A and the problem seams gone…

But I want them to be in separated stack (like before)…

hope this helps figuring the problem

thanks

3

I was working fine in version 0.56…

4

This is as a result of the changes in 0.63.0, i ran across the same issue. You no longer need to map an external service, if you remove that, you can then address stack/service directly through DNS so for instance serviceName.stack.rancher.internal

5

@Rucknar could you share the compose file of your app where you hit the issue? Named links should continue to work within the same stack as well as cross stacks

6

@Sebastien_Allamand could you share your original compose file as well? where services were linked cross stacks

7

Also what image (or base image, like ubuntu, alpine, etc) is the container based on where nslookup works but curl doesn’t?

8

@alena, @vincent here the section of my compose file concerned

for the DB part ;

cloudsso-db-master:
  restart: always
  expose:
  - '3306'
  environment:
    CONTEXT: RANCHER
    MYSQL_ROOT_PASSWORD: ${MYSQL_PWD}
  labels:
    io.rancher.sidekicks: cloudsso-data-master
    io.rancher.scheduler.affinity:host_label: host=backend01
  hostname: cloudsso-db-master
  image: ${CLOUDSSO_REGISTRY}/cloudsso-qualif/db-master:${CLOUDSSO_IMAGE_VERSION_DB_MASTER}
  volumes_from:
  - cloudsso-data-master
  volumes:
  - /sys/fs/cgroup:/sys/fs/cgroup:ro
cloudsso-data-master:
  image: busybox
  net: none
  command: /bin/true
  labels:
    io.rancher.container.start_once: 'true'
  hostname: cloudsso-data-master
  volumes:
  - /var/lib/mysql/cloudsso-db:/var/lib/mysql
  - /var/log/mysql/cloudsso-db:/var/log/mysql
  - /var/tmp/mysql/cloudsso-db:/var/tmp/mysql

And from the front from which I get connectivity issue ;

cloudsso-northapi:
  restart: always
  expose:
  - 80
  environment:
    CONTEXT: RANCHER
    ENV_NORTHAPI_MYSQL_LOGIN: ${CLOUDSSO_NORTHAPI_MYSQL_LOGIN}
    ENV_NORTHAPI_MYSQL_PASSWORD: ${CLOUDSSO_NORTHAPI_MYSQL_PASSWORD}
    ENV_DOMAIN: ${CLOUDSSO_DOMAIN}
  labels:
    io.rancher.scheduler.affinity:host_label: type=backend
    io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
  hostname: cloudsso-northapi
  image: ${CLOUDSSO_REGISTRY}/cloudsso-qualif/northapi:${CLOUDSSO_IMAGE_VERSION_NORTHAPI}
  external_links:
  - cloudsso-db/cloudsso-db-master:cloudsso-db-master

@vincent, my base images are centos7

sebastien

9

@Sebastien_Allamand I’ve deployed a simplified version of your application (no volumes, no user defined env vars, used centos:latest image) on Rancher v1.0.0-rc2, and was able to resolve cloudsso-db-master successfully from cloudsso-northapi. I’ve tested it with ping/dig.

We’ve put some fixes for Rancher internal-dns since Rancher 63.1 (https://github.com/rancher/cattle/pull/1514); would you mind giving it another try on rc2? Let me know if you still hit the problem, and we will debug further. Also could you share if your client uses ping for dns resolution, or something else?

10

Hi @alena,

The problem has disapear with new 1.0 version of Rancher.
Thanks a lot!

11

@Sebastien_Allamand anytime, let us know if you hit any other issues