Can't "exec" into pods, where rancher is running behind an NGINX reverse proxy, with TLS termination

I have a Nginx reverse proxy, which proxies traffic into my (Istio) ingress-gateway’s loadbalancer. And TLS termination is happening at this nginx. And DNS points to nginx’s ip. Rancher is exposed through this Ingress-gateway, Its working well and good, I can access rancher-ui and all. Also have proper ssl certs for the TLS. I can even download kubeconfig file from rancher ui and access the general kubectl commands also…

Except I can’t use exec or port-forward commands.
How to solve this?
Any help is much appreciated.

Heres my take on this:

  • I found another forum post (and docs) that have a similar configuration as mine. But their verbose output is different. I seem to be getting a 403 error from my nginx server, while exec.
  • I doubled checked Connection "Upgrade" upper case for websocket. Looks fine.
  • My cluster is rke cluster. And with that kubeconfig file that it gives, where I directly access the k8s server without a proxy in the middle. Then I am able to run exec commands fine. So its definitely not a problem with k8s.
  • I cannot seem to debug if the problem is with my nginx conf (or) with Istio ingress-gateway (or) with rancher (somehow). (But my 403 error is coming from nginx server only, from the logs below, so the problem might be with nginx reverse proxy only and not with ingress …(?) )
  • I tried changing the ip values in my rancher kubeconfig file, to directly reach the ingress loadbalancer, instead of nginx ip, to see if the problem is with nginx or not. But it doesn’t work that way because my ingressgateway only accepts http. (And I cannot force http in kubeconfig file, it says it requires some token/auth or something like that)
  • Right now I am the cluster owner and admin in rancher. Do I need to enable some more permissions somewhere in rancher. Never came across anything like that.

Here’s the output of the commands:

$ kubectl exec -it busybox-curl -- bash
Error from server:
$ 
$ kubectl -v=9 exec -it busybox-curl -- bash
I0819 23:55:28.694769    4761 loader.go:372] Config loaded from file:  <mask>.config
I0819 23:55:28.704348    4761 round_trippers.go:435] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.21.1 (<mask>) kubernetes/<mask>" -H "Authorization: Bearer <masked>" 'https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl'
I0819 23:55:29.440077    4761 round_trippers.go:454] GET https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl 200 OK in 735 milliseconds
I0819 23:55:29.440150    4761 round_trippers.go:460] Response Headers:
I0819 23:55:29.440167    4761 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: <mask>
I0819 23:55:29.440179    4761 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: <mask>
I0819 23:55:29.440189    4761 round_trippers.go:463]     Cache-Control: no-cache, private
I0819 23:55:29.440198    4761 round_trippers.go:463]     Date: Thu, 19 Aug 2021 18:25:29 GMT
I0819 23:55:29.440208    4761 round_trippers.go:463]     Content-Type: application/json
I0819 23:55:29.440216    4761 round_trippers.go:463]     Connection: keep-alive
I0819 23:55:29.440224    4761 round_trippers.go:463]     Audit-Id: <mask>
I0819 23:55:29.440234    4761 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0819 23:55:29.440241    4761 round_trippers.go:463]     X-Envoy-Upstream-Service-Time: 71
I0819 23:55:29.440249    4761 round_trippers.go:463]     Server: nginx/1.18.0 (Ubuntu)
I0819 23:55:29.440387    4761 request.go:1123] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"busybox-curl","namespace":"default","uid":"<mask>","resourceVersion":"<mask>","creationTimestamp":"2021-08-16T12:13:20Z","annotations":{"cni.projectcalico.org/podIP":"<mask>/32","cni.projectcalico.org/podIPs":"<mask>/32","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"busybox-curl\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"args\":["<mask>"],\"image\":\"<mask>\",\"name\":\"busybox-curl\"}]}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2021-08-16T12:13:20Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"busybox-curl\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2021-08-16T12:13:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-08-16T12:13:32Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"<mask>\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-fh7k5","secret":{"secretName":"default-token-fh7k5","defaultMode":420}}],"containers":[{"name":"busybox-curl","image":"<mask>","args":["<mask>"],"resources":{},"volumeMounts":[{"name":"default-token-fh7k5","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker7","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:20Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:32Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:32Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:20Z"}],"hostIP":"<mask>","podIP":"<mask>","podIPs":[{"ip":"<mask>"}],"startTime":"2021-08-16T12:13:20Z","containerStatuses":[{"name":"busybox-curl","state":{"running":{"startedAt":"2021-08-16T12:13:31Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"<mask>","imageID":"<mask>","containerID":"<mask>","started":true}],"qosClass":"BestEffort"}}
I0819 23:55:29.448282    4761 podcmd.go:88] Defaulting container name to busybox-curl
I0819 23:55:29.448774    4761 round_trippers.go:435] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.21.1 (<mask>) kubernetes/<mask>" -H "Authorization: Bearer <masked>" 'https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl/exec?command=bash&container=busybox-curl&stdin=true&stdout=true&tty=true'
                                                                                                                                                                                           I0819 23:55:29.823389    4761 round_trippers.go:454] POST https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl/exec?command=bash&container=busybox-curl&stdin=true&stdout=true&tty=true 403 Forbidden in 374 milliseconds
                                                                                            I0819 23:55:29.823443    4761 round_trippers.go:460] Response Headers:
                                                                                                                                                                  I0819 23:55:29.823464    4761 round_trippers.go:463]     Content-Length: 0
                                                                                                                                                                                                                                            I0819 23:55:29.823474    4761 round_trippers.go:463]     Connection: keep-alive
                                                                                                                                                                                                                                                                                                                           I0819 23:55:29.823519    4761 round_trippers.go:463]     Server: nginx/1.18.0 (Ubuntu)
                                     I0819 23:55:29.823533    4761 round_trippers.go:463]     Date: Thu, 19 Aug 2021 18:25:29 GMT
                                                                                                                                 I0819 23:55:29.824484    4761 helpers.go:216] server response object: [{
  "metadata": {}
}]
F0819 23:55:29.824523    4761 helpers.go:115] Error from server: 
goroutine 1 [running]:
<traceback traceback>
$

Here’s my nginx.conf file

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
		worker_connections 768;
		# multi_accept on;
}

http {

		##
		# Basic Settings
		##

		sendfile on;
		tcp_nopush on;
		tcp_nodelay on;
		keepalive_timeout 65;
		types_hash_max_size 2048;
		# server_tokens off;

		# server_names_hash_bucket_size 64;
		# server_name_in_redirect off;

		include /etc/nginx/mime.types;
		default_type application/octet-stream;

		##
		# SSL Settings
		##

		ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
		ssl_prefer_server_ciphers on;

		##
		# Logging Settings
		##

		access_log /var/log/nginx/access.log;
		error_log /var/log/nginx/error.log;

		##
		# Gzip Settings
		##

		gzip on;

		# gzip_vary on;
		# gzip_proxied any;
		# gzip_comp_level 6;
		# gzip_buffers 16 8k;
		# gzip_http_version 1.1;
		# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

		##
		# Virtual Host Configs
		##

		#include /etc/nginx/conf.d/*.conf;
		#include /etc/nginx/sites-enabled/*;

		ssl_certificate <ssl-certificate>;
		ssl_certificate_key <ssl-certificate-key>;

		server{
				listen <ip-0>:443 ssl;
				server_name <rancher-hostname>;

				location / {
						proxy_pass                      http://<rancher-ingress-gateway-b>;
						proxy_http_version              1.1;
						proxy_set_header                Upgrade $http_upgrade;
						proxy_set_header                Connection "Upgrade";
						proxy_set_header                Host $host;
						proxy_set_header                Referer $http_referer;
						proxy_set_header                X-Real-IP $remote_addr;
						proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
						proxy_set_header                X-Forwarded-Proto $scheme;
						proxy_pass_request_headers      on;
				}
		}
		server{
				listen <ip-0>:443 ssl default_server;

				location / {
						proxy_pass                      http://<other-lb-1>;
						proxy_http_version              1.1;
						proxy_set_header                Upgrade $http_upgrade;
						proxy_set_header                Connection "Upgrade";
						proxy_set_header                Host $host;
						proxy_set_header                Referer $http_referer;
						proxy_set_header                X-Real-IP $remote_addr;
						proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
						proxy_set_header                X-Forwarded-Proto $scheme;
						proxy_pass_request_headers      on;
				}
		}
		server{
				listen <ip-1>:443 ssl;

				location / {
						proxy_pass                      http://<other-lb-2>;
						proxy_http_version              1.1;
						proxy_set_header                Upgrade $http_upgrade;
						proxy_set_header                Connection "Upgrade";
						proxy_set_header                Host $host;
						proxy_set_header                Referer $http_referer;
						proxy_set_header                X-Real-IP $remote_addr;
						proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
						proxy_set_header                X-Forwarded-Proto $scheme;
						proxy_pass_request_headers      on;
				}
		}
}
stream {
		server{
				listen <ip-0>:5432;
				proxy_pass <other-lb-1>:5432;
		}
}

Rancher version : 2.5.9
RKE version: 1.2.8
K8s version: 1.20.6
kubectl version: 1.21.1-darwin

I got the same issue. But no solution found. If you found any solution, please share

Hi, I found a solution to this.
Commands like kubectl exec (and attach, port-forward) use the deprecated SPDY protocol. SPDY requires TLS, so if the load balancer proxy server (NGINX) uses TLS termination, the connection cannot be upgraded at the Rancher proxy service.
To fix this, I added the following annotations to Rancher ingress:

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

and set the backend port number to 443, the Rancher HTTPS port.
This will allow TLS redirection to the Rancher backend, and the SPDY protocol will persist.

Thanks @Anders_Swanson . Turns out SDPY is the problem but might not be related to TLS passthrough.
As I said I was using Istio Ingress (which is based on envoy) which doesnt support connection upgrades to SDPY. Relevant github issues:

In a nutshell with Istio Ingress to Rancher, exec port-forward or similar commands are not possible (yet).

I switched back to nginx Ingress, And everything is working, even without TLS Passthrough (i.e., TLS termination at nginx).

Thanks

This is an ancient thread, but I solved this issue (at least for Istio) and wanted to provide what I had for those who might run into this. You need to create a cluster-wide EnvoyFilter to add support for the SPDY upgrade:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: rancher-upgrade-filter
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:      
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          "upgradeConfigs": [ { "upgradeType": "SPDY/3.1" } ]

This should resolve the exec issue.

Thank you @adam.toy . This is now relevant to me again after 1 year of your reply :sweat_smile:.
Your EnvoyFilter for the upgrade works for me. (Although the HttpConnectionManager, in the latest version, is now type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.)
I am able to access logs, perform exec, etc.