Single Node Rke Rancher inconsistency goes down

1

I have created rancher with a single node using the minimum cluster.yml as given in rke documentation. Example Cluster.ymls | RKE1

I have provisioned a new cluster with 2 machines (master and worker). I observed that the Rancher UI inconsistently goes down. I tried to check the logs and found out that there was some problem with the cert manager initially for which normal docker restart fixed my problem.

Now I observed that the UI is down and the docker ps is showing only 8 containers up and running all the other containers are existing.
No kubectl related commands to get the cluster information are working.
No helm commands are working.

image
image1494×430 27.6 KB

I am new to Rancher and I am trying to understand this behaviour. Any help is appreciated

2

Hi,

Can you show the etcd log using
docker logs -f etcd

4

Hi,
Thank you for your reply
Here are the logs

{"level":"info","ts":"2023-10-02T19:12:41.846Z","caller":"embed/etcd.go:478","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/ssl/kube-etcd-172-27-16-68.pem, key = /etc/kubernetes/ssl/kube-etcd-172-27-16-68-key.pem, client-cert=, client-key=, trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"]}
{"level":"info","ts":"2023-10-02T19:12:41.847Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]}
{"level":"info","ts":"2023-10-02T19:12:41.847Z","caller":"embed/etcd.go:307","msg":"starting an etcd server","etcd-version":"3.5.0","git-sha":"946a5a6f2","go-version":"go1.16.3","go-os":"linux","go-arch":"amd64","max-cpu-set":6,"max-cpu-available":6,"member-initialized":true,"name":"etcd-172.27.16.68","data-dir":"/var/lib/rancher/etcd/","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/etcd/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://172.27.16.68:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://172.27.16.68:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
panic: freepages: failed to get all reachable pages (page 1162: multiple references)

goroutine 162 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2(0xc0000943c0)
        /home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/go.etcd.io/bbolt@v1.3.6/db.go:1056 +0xe9
created by go.etcd.io/bbolt.(*DB).freepages
        /home/remote/sbatsche/.gvm/pkgsets/go1.16.3/global/pkg/mod/go.etcd.io/bbolt@v1.3.6/db.go:1054 +0x1cd
5

Out of memory or out-of-diskspace…

6

How can i restore this?

7

Am facing exactly same issue , but i do have enough disk space .Getting the same logs as omkar