I have a Nginx reverse proxy, which proxies traffic into my (Istio) ingress-gateway’s loadbalancer. And TLS termination is happening at this nginx. And DNS points to nginx’s ip. Rancher is exposed through this Ingress-gateway, Its working well and good, I can access rancher-ui and all. Also have proper ssl certs for the TLS. I can even download kubeconfig file from rancher ui and access the general kubectl commands also…
Except I can’t use exec or port-forward commands.
How to solve this?
Any help is much appreciated.
Heres my take on this:
- I found another forum post (and docs) that have a similar configuration as mine. But their verbose output is different. I seem to be getting a 403 error from my nginx server, while
exec. - I doubled checked
Connection "Upgrade"upper case for websocket. Looks fine. - My cluster is rke cluster. And with that kubeconfig file that it gives, where I directly access the k8s server without a proxy in the middle. Then I am able to run
execcommands fine. So its definitely not a problem with k8s. - I cannot seem to debug if the problem is with my nginx conf (or) with Istio ingress-gateway (or) with rancher (somehow). (But my 403 error is coming from nginx server only, from the logs below, so the problem might be with nginx reverse proxy only and not with ingress …(?) )
- I tried changing the ip values in my rancher kubeconfig file, to directly reach the ingress loadbalancer, instead of nginx ip, to see if the problem is with nginx or not. But it doesn’t work that way because my ingressgateway only accepts http. (And I cannot force http in kubeconfig file, it says it requires some token/auth or something like that)
- Right now I am the cluster owner and admin in rancher. Do I need to enable some more permissions somewhere in rancher. Never came across anything like that.
Here’s the output of the commands:
$ kubectl exec -it busybox-curl -- bash
Error from server:
$
$ kubectl -v=9 exec -it busybox-curl -- bash
I0819 23:55:28.694769 4761 loader.go:372] Config loaded from file: <mask>.config
I0819 23:55:28.704348 4761 round_trippers.go:435] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.21.1 (<mask>) kubernetes/<mask>" -H "Authorization: Bearer <masked>" 'https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl'
I0819 23:55:29.440077 4761 round_trippers.go:454] GET https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl 200 OK in 735 milliseconds
I0819 23:55:29.440150 4761 round_trippers.go:460] Response Headers:
I0819 23:55:29.440167 4761 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: <mask>
I0819 23:55:29.440179 4761 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: <mask>
I0819 23:55:29.440189 4761 round_trippers.go:463] Cache-Control: no-cache, private
I0819 23:55:29.440198 4761 round_trippers.go:463] Date: Thu, 19 Aug 2021 18:25:29 GMT
I0819 23:55:29.440208 4761 round_trippers.go:463] Content-Type: application/json
I0819 23:55:29.440216 4761 round_trippers.go:463] Connection: keep-alive
I0819 23:55:29.440224 4761 round_trippers.go:463] Audit-Id: <mask>
I0819 23:55:29.440234 4761 round_trippers.go:463] X-Content-Type-Options: nosniff
I0819 23:55:29.440241 4761 round_trippers.go:463] X-Envoy-Upstream-Service-Time: 71
I0819 23:55:29.440249 4761 round_trippers.go:463] Server: nginx/1.18.0 (Ubuntu)
I0819 23:55:29.440387 4761 request.go:1123] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"busybox-curl","namespace":"default","uid":"<mask>","resourceVersion":"<mask>","creationTimestamp":"2021-08-16T12:13:20Z","annotations":{"cni.projectcalico.org/podIP":"<mask>/32","cni.projectcalico.org/podIPs":"<mask>/32","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"busybox-curl\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"args\":["<mask>"],\"image\":\"<mask>\",\"name\":\"busybox-curl\"}]}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2021-08-16T12:13:20Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"busybox-curl\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2021-08-16T12:13:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-08-16T12:13:32Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"<mask>\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-fh7k5","secret":{"secretName":"default-token-fh7k5","defaultMode":420}}],"containers":[{"name":"busybox-curl","image":"<mask>","args":["<mask>"],"resources":{},"volumeMounts":[{"name":"default-token-fh7k5","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker7","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:20Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:32Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:32Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-08-16T12:13:20Z"}],"hostIP":"<mask>","podIP":"<mask>","podIPs":[{"ip":"<mask>"}],"startTime":"2021-08-16T12:13:20Z","containerStatuses":[{"name":"busybox-curl","state":{"running":{"startedAt":"2021-08-16T12:13:31Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"<mask>","imageID":"<mask>","containerID":"<mask>","started":true}],"qosClass":"BestEffort"}}
I0819 23:55:29.448282 4761 podcmd.go:88] Defaulting container name to busybox-curl
I0819 23:55:29.448774 4761 round_trippers.go:435] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.21.1 (<mask>) kubernetes/<mask>" -H "Authorization: Bearer <masked>" 'https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl/exec?command=bash&container=busybox-curl&stdin=true&stdout=true&tty=true'
I0819 23:55:29.823389 4761 round_trippers.go:454] POST https://<mask>/k8s/clusters/c-wgq28/api/v1/namespaces/default/pods/busybox-curl/exec?command=bash&container=busybox-curl&stdin=true&stdout=true&tty=true 403 Forbidden in 374 milliseconds
I0819 23:55:29.823443 4761 round_trippers.go:460] Response Headers:
I0819 23:55:29.823464 4761 round_trippers.go:463] Content-Length: 0
I0819 23:55:29.823474 4761 round_trippers.go:463] Connection: keep-alive
I0819 23:55:29.823519 4761 round_trippers.go:463] Server: nginx/1.18.0 (Ubuntu)
I0819 23:55:29.823533 4761 round_trippers.go:463] Date: Thu, 19 Aug 2021 18:25:29 GMT
I0819 23:55:29.824484 4761 helpers.go:216] server response object: [{
"metadata": {}
}]
F0819 23:55:29.824523 4761 helpers.go:115] Error from server:
goroutine 1 [running]:
<traceback traceback>
$
Here’s my nginx.conf file
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
#include /etc/nginx/conf.d/*.conf;
#include /etc/nginx/sites-enabled/*;
ssl_certificate <ssl-certificate>;
ssl_certificate_key <ssl-certificate-key>;
server{
listen <ip-0>:443 ssl;
server_name <rancher-hostname>;
location / {
proxy_pass http://<rancher-ingress-gateway-b>;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header Referer $http_referer;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass_request_headers on;
}
}
server{
listen <ip-0>:443 ssl default_server;
location / {
proxy_pass http://<other-lb-1>;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header Referer $http_referer;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass_request_headers on;
}
}
server{
listen <ip-1>:443 ssl;
location / {
proxy_pass http://<other-lb-2>;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header Referer $http_referer;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass_request_headers on;
}
}
}
stream {
server{
listen <ip-0>:5432;
proxy_pass <other-lb-1>:5432;
}
}
Rancher version : 2.5.9
RKE version: 1.2.8
K8s version: 1.20.6
kubectl version: 1.21.1-darwin