AWS EKS cluster import to rancher fails

I deployed the rancher container behind Nginx using docker-compose,

version: ‘3.3’
services:
rancher:
image: rancher/rancher:v2.6.7
hostname: rancher
restart: unless-stopped
privileged: true
volumes:
- /opt/rancher:/var/lib/rancher
networks:
- nginx
networks:
nginx:
external: true

I was able to successfully log in to rancher. I imported the EKS cluster. But when I click on the cluster, I get an error like this in browser,

HTTP Error 500: Internal Server Error {“data”:“invalid upgrade response: status code 200”}

Cattle-cluster agent and metrics server run fine. I got some error logs from the rancher container,

[![W1018 13:25:29.573852 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.RoleBinding ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.573983 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.APIService ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.574214 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.ServiceAccount ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.574310 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Namespace ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575083 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Secret ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575115 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Node ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575130 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Role ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575144 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.LimitRange ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575159 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.ClusterRoleBinding ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575211 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.ClusterRole ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.575230 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.ConfigMap ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
W1018 13:25:29.574910 33 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.ResourceQuota ended with: an error on the server (“unable to decode an event from the watch stream: tunnel disconnect”) has prevented the request from succeeding
2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60764 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60774 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60788 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60790 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:25:49 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:54436 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:25:49 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:54438 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:25:49 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:54444 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:25:49 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:54458 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:25:59 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49482 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:25:59 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49496 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:00 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49500 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:26:00 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49506 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
E1018 13:26:10.501981 33 upgradeaware.go:338] Proxy upgrade error: invalid upgrade response: status code 201
2022/10/18 13:26:10 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60272 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:26:10 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60282 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:10 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60284 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:26:10 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60290 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:21 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:53034 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:26:21 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:53038 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:21 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:53050 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:26:21 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:53056 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:32 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49084 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:26:32 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49090 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:32 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49098 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:26:32 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49110 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:42 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:58782 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:26:42 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:58784 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:42 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:58796 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:26:43 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:58810 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:53 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49124 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:26:53 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49132 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:26:53 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49144 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:26:53 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49152 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:04 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:59552 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:27:04 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:59558 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:27:04 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:59562 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:04 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:59566 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:05 [ERROR] Unknown error: Operation cannot be fulfilled on preferences.management.cattle.io “last-visited”: the object has been modified; please apply your changes to the latest version and try again
2022/10/18 13:27:05 [ERROR] Unknown error: Operation cannot be fulfilled on preferences.management.cattle.io “last-visited”: the object has been modified; please apply your changes to the latest version and try again
E1018 13:27:06.133118 33 upgradeaware.go:338] Proxy upgrade error: invalid upgrade response: status code 200
2022/10/18 13:27:08 [ERROR] Error during subscribe websocket: close sent
E1018 13:27:10.574539 33 upgradeaware.go:338] Proxy upgrade error: invalid upgrade response: status code 201
E1018 13:27:12.312189 33 upgradeaware.go:338] Proxy upgrade error: invalid upgrade response: status code 200
2022/10/18 13:27:14 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:34594 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:27:15 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:34600 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:27:15 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:34602 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:15 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:34608 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:25 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:52654 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:27:25 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:52660 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:27:25 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:52666 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:26 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:52680 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
E1018 13:27:33.158230 33 upgradeaware.go:338] Proxy upgrade error: invalid upgrade response: status code 200
2022/10/18 13:27:35 [ERROR] Error during subscribe websocket: close sent
2022/10/18 13:27:36 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:44344 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:27:36 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:44358 (X-Forwarded-For: 18.141.153.73): response 400: cluster not found
2022/10/18 13:27:36 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:44362 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:27:36 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:44418 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found

Can anyone tell me where did I go wrong,

Looks like you might be missing proxy pass configuration. Take a look at the example config here and see if that helps:

That looks like a bug. Open a ticket with support

2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60764 (X-Forwarded-For: 54.151.198.109): response 400: cluster not found
2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:60774 (X-Forwarded-For: 18.138.249.253): response 400: cluster not found
2022/10/18 13:25:38 [ERROR] Failed to handle tunnel request from remote address