Rancher 2.6 on Single node Kubernetes cluster stuck with replicas and not coming up

1

Hi all,

I setup Rancher 2.6 on a single node kubernetes cluster using this link : 3. Install Rancher | Rancher
I am pretty new to kubernetes.

We have a corporate proxy and everything worked fine till installation.I setup rke,helm and installed cert-manager via helm. Next I proceeded to install rancher.

kubectl create namespace cattle-system

#Install Rancher
helm upgrade --install rancher rancher-latest/rancher
–namespace cattle-system
–set bootstrapPassword=“dummy335”
–set hostname=ln228drcpsvr01.swcdev
–set proxy=http://user:password@10.10.61.8:8080/
–set no_proxy=127.0.0.0/8\,10.0.0.0/8\,cattle-
system.svc\,172.16.0.0/12\,192.168.0.0/16\,.svc\,.cluster.local

Output

WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: kube_config_rancher-cluster.yaml
Release “rancher” does not exist. Installing it now.
NAME: rancher
LAST DEPLOYED: Tue Mar 1 13:58:47 2022
NAMESPACE: cattle-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Rancher Server has been installed.

NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up.

Check out our docs at Rancher Docs: Rancher Brand Guidelines & Resources

If you provided your own bootstrap password during installation, browse to https://linuxserver.domain to get started.

But I cannot access the UI.

I tried seeing the deployment status

kubectl rollout status deployment -n cattle-system rancher

Waiting for deployment “rancher” rollout to finish: 0 of 3 updated replicas are available…
Waiting for deployment “rancher” rollout to finish: 1 of 3 updated replicas are available…
Waiting for deployment “rancher” rollout to finish: 2 out of 3 new replicas have been updated…
Waiting for deployment “rancher” rollout to finish: 1 of 3 updated replicas are available…
Waiting for deployment “rancher” rollout to finish: 2 of 3 updated replicas are available…
Waiting for deployment “rancher” rollout to finish: 2 out of 3 new replicas have been updated…
Waiting for deployment “rancher” rollout to finish: 1 of 3 updated replicas are available…
Waiting for deployment “rancher” rollout to finish: 2 out of 3 new replicas have been updated…
Waiting for deployment “rancher” rollout to finish: 0 of 3 updated replicas are available…

Any idea whats the issue here.

There are containers running in the system.

image
image1916×364 32.9 KB

kubectl get pods --all-namespaces returns this.

image
image1174×288 13.7 KB

2

Did you already check out the basics?

3

Yea I did. I think the issue is with last step where rancher installation happens. The pods are in evicted state.

certmanager installation was success and pods are running. Just the rancher pods are getting into evicted state.

4

I debugged the issue and fixed it. Turns out that device mapper storage driver caused the issue.
Pods kept restarting continuously. This is why I couldn’t access Rancher.

I have since changed to overlay2 with docker recommended settings and it started working fine.

Thanks,
Vishnu

5

Thanks for sharing the actual issue + solution. :slightly_smiling_face:

6

I’m getting this loop, but I dont’ understand how to fix it. I installed K3s via ansible using this tutorial The FASTEST Way to run Kubernetes at Home - YouTube
And now im installing rancher in that k3s installation but getting this loop:

Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
7

i’m having this exact issue and found where you do the overlay2 in docker (Use the OverlayFS storage driver | Docker Documentation), however I’m using containerd. Is it possibly same issue?

8

For me this issue was caused from not having enough storage to download the container during deployment, I was unable to get logs for the pods but you can see the error if you do kubectl describe pod rancher-<ID> -n cattle-system for pods as they try and come up.