Reinstall causes x509: certificate signed by unknown authority

k3s, k3OS, and k3d
1

I need to know how to properly clean-up after uninstalling HA k3s so I don’t get this error: x509: certificate signed by unknown authority

Steps to reproduce:

  1. Uninstall nodes

    sudo /usr/local/bin/k3s-uninstall.sh
    sudo /usr/local/bin/k3s-agent-uninstall.sh

  2. Clear the DB

    use k3sdb;
    drop table kine;

  3. Reinstall the first master node.

  4. Try to check status:

    $ sudo k3s kubectl get nodes
    Unable to connect to the server: x509: certificate signed by unknown authority

I did not set a tls-san, is that required? Should it equal one of the master host names, or can it be arbitrary, like master-lan. I wasn’t sure how to use it.

Here is how I ran in the install:

curl -sfL [get k3s  link] | sh -s - server --datastore-endpoint="mysql://k3s-admin:""redacted""@tcp(k3s-db-server:3306)/k3sdb"

export NODE_TOKEN=REDACTED::node::REDACTED
curl -sfL [get k3s link]  agent | INSTALL_K3S_EXEC="agent --server https://192.168.1.10:6443 --token $NODE_TOKEN" sh -

What am I missing?

3

I don’t think this is a “cleanup” issue. Before switching to an external database all three master nodes showed available. With an external database everything works fine with the first two nodes. They both have the same node token, and kubectl commands work from both.

When I added the third master using the same command, it gets a different node token, and first two masters start getting the “x509: certificate signed by unknown authority” message when trying to use kubectl. On the third master, I can use kubectl, but k3s kubectl get nodes shows the first two masters in the list but not the third where the command is run.

Does this have something to do with the third master becoming a registration node? Is that why the node-token is different than the other two nodes?

4

It was a clean-up issue. I re-imaged all the notes and then I was able to get all three master nodes up and running against an external MySQL database.

I should not have to do that. The uninstall script removes all the k3s software and directories, but it does not level the servers in a good state for reinstalling k3s. Something is left behind regarding 509 certificates. I don’t know enough about certificates know how to clean that up. I ran

sudo dpkg-reconfigure ca-certificates

That broke install k3s completely, thus the re-imaging, which fixed the problem, but should not be necessary. What should I have done to avoid re-imaging before re-installing k3s.

5

Hi @smitchell, same issue here … any advise?

Best

6

Hi guys, same issue here, any help besides reimaging the boxes?

7

This is due to previous .kube/config not being deleted.

Delete .kube/config and reinstall fix the problem.

8

You don’t even have to reinstall. (My preference was to move my config to config.bak). Shutdown the app with no config file and restart it. It will then create a new, clean config with the correct certs. (To be clear, I’m not running K3d. I was running Rancher Desktop).