See github issue here: Kubernetes dashboard link returns 404 · Issue #8128 · rancher/rancher · GitHub
Bottom line. Our cluster is up, but we can’t connect to it and we can’t manage it. Can’t connect via kubectl. Can’t see dashboard (returns 404.) This happens across all 4 of our k8s environments on Rancher 1.4.1 and rancher/k8s:v1.5.2-rancher1-4.
This started happening without any changes to our environment about 7 hours ago. We are basically in disaster recovery mode now.
We have tried:
- Restarting rancher server
- Restarting k8s infrastructure stack (except for etcd)
- Many other things that probably didn’t matter.
Each host in the cluster is still up. Containers are still running. Kubernetes infrastructure stack page is all green.
Any ideas?
@ryanwalls, we have seen the same thing. Ours was inconsistent (sometimes we’d see the 404, sometimes the dashboard would load successfully) . I see on the github issue you switched from ALB to ELB. We have been using an ALB also - But, we originally tried using an ELB (per the rec from Rancher) with the ProxyProtocol mode enable, but we still saw the rancher-agent failing to communicate back to rancher-server over websockets. (Which, I think is the point of having enabled ProxyProtocol)
So you were able to make an ELB work with rancher-server?
Did you just follow the doc here? https://docs.rancher.com/rancher/v1.2/en/installing-rancher/installing-server/basic-ssl-config/#running-rancher-server-behind-an-elastic-load-balancer-elb-in-aws-with-ssl
Any other hints or observations?
