Timeout on kubectl exec on with nginx ssl termination

Archive Rancher 1.x
1

I’m trying to debug an issue using kubectl and Rancher. I’ve upgraded our Rancher setup to 1.2.0-pre2 because I’d like to use kubectl exec. But when I run it, it’ll get a timeout error. I’ve tried running on the same machine as I have Rancher running and I get the same error. I can certainly do kubectl exec from within the UI kubectl shell and other kubectl commands work fine.

The only thing I can think of is that we have an nginx ssl termination in front of rancher, as per the official instructions. I’ve tried playing around with the nginx config, adding support for http2 and so on but no luck.

Any suggestions welcome?

Snippet from verbose output from kubectl (–v=9):

I0907 09:09:52.464221 1 exec.go:179] defaulting container name to mongo I0907 09:09:52.464789 1 round_trippers.go:299] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true I0907 09:09:52.465308 1 round_trippers.go:299] curl -k -v -XPOST -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" -H "Authorization: Basic OTU1OUVDRTE5QjUwQUIxQkU3NzQ6VnRHTUdUQ3lZU0d6Tm85OXB3Z0FaNXB5c2RkeWgxckoyNnlBNlFTZQ==" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true I0907 09:09:52.639324 1 round_trippers.go:318] POST https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true 101 Switching Protocols in 173 milliseconds I0907 09:09:52.640316 1 round_trippers.go:324] Response Headers: I0907 09:09:52.640783 1 round_trippers.go:327] Server: nginx/1.9.15 I0907 09:09:52.641125 1 round_trippers.go:327] Date: Wed, 07 Sep 2016 09:09:47 GMT I0907 09:09:52.641484 1 round_trippers.go:327] Connection: upgrade I0907 09:09:52.641725 1 round_trippers.go:327] Upgrade: SPDY/3.1 I0907 09:09:52.642070 1 round_trippers.go:327] X-Stream-Protocol-Version: v2.channel.k8s.io I0907 09:09:52.642303 1 round_trippers.go:318] POST https://rancher.dev.stpl.stormtest.com/r/projects/1a8/kubernetes/api/v1/namespaces/default/pods/mongo-kwf8k/exec?command=ls&command=C%3A%2FProgram&command=Files%2FGit&container=mongo&container=mongo&stderr=true&stdout=true&tty=true 101 Switching Protocols in 177 milliseconds I0907 09:09:52.642738 1 round_trippers.go:324] Response Headers: I0907 09:09:52.643155 1 round_trippers.go:327] X-Stream-Protocol-Version: v2.channel.k8s.io I0907 09:09:52.643580 1 round_trippers.go:327] Server: nginx/1.9.15 I0907 09:09:52.643958 1 round_trippers.go:327] Date: Wed, 07 Sep 2016 09:09:47 GMT I0907 09:09:52.644205 1 round_trippers.go:327] Connection: upgrade I0907 09:09:52.644501 1 round_trippers.go:327] Upgrade: SPDY/3.1 F0907 09:10:22.657095 1 helpers.go:108] error: Timeout occured

2

Small update to this. I spun up a second rancher instance without nginx ssl termination and I was able to run remove commands. It’s definitely the nginx configuration.

3

Found the problem and it’s frigg’in dumb. For kubectl to work “upgrade” needs to be capitalized:

proxy_set_header Connection “Upgrade”;

So your nginx configuration should be:

upstream rancher {
server rancher-server:8080;
}

server {
listen 443 ssl;
server_name ;
ssl_certificate <cert_file>;
ssl_certificate_key <key_file>;

location / {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://rancher;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
    proxy_read_timeout 900s;
}

}

server {
listen 80;
server_name ;
return 301 https://$server_name$request_uri;
}

4

OMG. You’re my life saver. Such a small and annoying little thing! I was using the jwilder_nginx proxy, and it has upgrade configured by default; but not Upgrade.

5

Related: https://github.com/rancher/rancher/issues/9155 . The header is supposed to be treated as case-insensitive, so this is a bug in k8s. I don’t know that that person ever put a bug in for k8s if you’d like to do so yourself. You could also probably put in a 1 byte PR for jwilder/nginx-proxy to make it capital by default, since that is much more common than lowercase.