Why am I getting a message that it can't log in when trying to use kubectl

1

I have a running k3s server that has worked great so I haven’t had to access it. Now I would like to change something but when I ssh into the server and run KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get pods -A I get error: You must be logged in to the server (Unauthorized). What am I missing?

2

I got the same issue here. Did you find the root cause?

Also when I try to access the k3s-cluster from the Rancher UI, everything seems to be okay, but it’s not possible to get logs or open a terminal on the containers. Rancher UI indicate that it is “Disconnected”.

When using the kubeconfig from Rancher, I get the following error with kubectl logs: “error: You must be logged in to the server (the server has asked for the client to provide credentials…”. Other commands like kubectl get ns work without problem.

3

@J_Gleason I found the issue. The certificates in my k3s were expired. A simple restart of the k3s service fixed the issue. See also Rancher Docs: Advanced Options and Configuration

4

We are having the same issue suddenly.

However, it appears our certificates are not expired:

$ openssl s_client -connect localhost:6443 -showcerts < /dev/null 2>&1 | openssl x509 -noout -enddate

notAfter=Nov  9 11:53:58 2023 GMT
5

I’ve started seeing the same issue a few days ago and I cannot figure the cause. Anyone has any clues. My certificates are also not expired.

6

I got the same issue and what was helpful for me is to:

  1. Restart the k3s service on host running k3s.
  2. Copy the k3s.yaml from host running k3s located at /etc/rancher/k3s/k3s.yaml to my client host to ~/.kube/config.json and I got the access to the cluster.
    The restarting of the k3s service made my Jenkins jobs running on the cluster - they was stacked before.
    The root cause is, for my understanding, that the k3s is renewing it’s certificates automatically and to make them effective you need to restart the k3s service (service k3s restart).
    After that to be able to run commands with kubectl you need to renew your certificates in config.json like I mentioned before.
7

Most errors are usually due to an error with the SSL certificate or network issues , so make sure those are functioning correctly.