Links not working during service upgrade

Rancher 1.x
1

I’m trying to upgrade a service but 50% of the times the links are not working, nginx in fact outputs:

11/10/2015 8:56:40 AM2015/11/10 16:56:40 [emerg] 1#1: host not found in upstream "api:8000" in /etc/nginx/nginx.conf:38
11/10/2015 8:56:40 AMnginx: [emerg] host not found in upstream "api:8000" in /etc/nginx/nginx.conf:38

after a couple of tries to upgrade it works fine, someone else had this issue?

2

Could you provide a docker-compose.yml example of a previous service (with the linked services)? And what you are upgrading to? I’d like to try what you’re doing.

Also, what commands are you using to upgrade? The UI or rancher-compose? Are you using the rancher-compose up --upgrade or rancher-compose upgrade?

3

I’ve created everything from the UI, this is the compose:

backend-prod:
  restart:
    maximumRetryCount: 2
    name: on-failure
  environment:
    DATABASE_URL: xxx
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: company/project-backend:master
  stdin_open: true
proxy-prod:
  restart:
    name: always
  environment:
    BACKEND_URL: http://api:8000/api/dataproxy
  tty: true
  image: company/project-proxy:master
  links:
  - backend-prod:api
  stdin_open: true
frontend-prod:
  restart:
    name: always
  labels:
    io.rancher.container.pull_image: always
  tty: true
  image: company/project-frontend:master
  links:
  - backend-prod:api
  - proxy-prod:proxy
  stdin_open: true

To upgrade I run the upgrade from the UI just to re-pull the image

4

Also, the error I get is from the frontend-prod service, an nginx based service with this configuration:

  upstream api {
    server api:8000;
  }

  upstream proxy {
    server proxy:9000;
  }

  location ~ /api/something/[^\/]+/data {
    proxy_pass http://proxy;
  }
5

Now, the service linked as api, doesn’t work at all! The frontend container can’t connect and creating new frontend containers shows that nginx error everytime!

All our services are offline because of that, and I’ve just upgraded to the latest rancher and rancher agents versions

Even when creating a new service

6

I’ve just tried also with service name in upstream configuration, same thing.
Changing to IP instead works fine

7

@denise any idea, something that I could try?

8

We’ve just updated the internal DNS service so would you be willing to upgrade to v0.46.0? I haven’t had the chance to try out your application, but would recommend looking into upgrading.

9

@denise I’ve already 0.46, in fact before upgrading links were working sometimes, after 0.46 they never had, with the link name (api in this case) and neither with the service name

10

Solution:

location /api {
    resolver 169.254.169.250;
    set $api "api";
    proxy_pass http://$api:8000;
}

in that way the backend dns is resolved dynamically and works after nginx booting

3 Likes
11

Where does IP 169.254.169.250 come from? I did not find it in docs.

12

It’s the ip of the network agent on each host IIRC. It’s the default dns server to provide container discovery

1 Like
13

FYI I’ve encountered the same problem, adding a delay with a sleep 5 before running the actuel nginx bin resolved all the dns problem.

14

Good to know, but still I think the dynamic dns works best, since it will also change in case the other containers gets destroyed/recreated/scaled

15

This is with the dynamic dns , just let 1 / 2 second so that the dns is fully setup and responding.

16

If you’re talking about the version with the variable, yes, otherwise, not, that’s not dynamic