Rancher UI down, error out with - Panic: freepages issue

Rancher 2.x
1

Hi,

Needs urgent help on Rancher. Rancher UI not coming up, it’s running as docker container and container is failing with below error.

2021-10-20 21:12:19.627803 I | embed: name = default

2021-10-20 21:12:19.627874 I | embed: data dir = management-state/etcd

2021-10-20 21:12:19.627910 I | embed: member dir = management-state/etcd/member

2021-10-20 21:12:19.627944 I | embed: heartbeat = 500ms

2021-10-20 21:12:19.627978 I | embed: election = 5000ms

2021-10-20 21:12:19.628012 I | embed: snapshot count = 100000

2021-10-20 21:12:19.628077 I | embed: advertise client URLs = http://localhost:2379

2021-10-20 21:12:19.628133 I | embed: initial advertise peer URLs = http://localhost:2380

2021-10-20 21:12:19.628187 I | embed: initial cluster =

panic: freepages: failed to get all reachable pages (page 2656: multiple references)

goroutine 162 [running]:

go.etcd.io/bbolt.(*DB).freepages.func2(0xc0000ac660)

/home/ec2-user/go/pkg/mod/go.etcd.io/bbolt@v1.3.3/db.go:1003 +0xe5

created by go.etcd.io/bbolt.(*DB).freepages

/home/ec2-user/go/pkg/mod/go.etcd.io/bbolt@v1.3.3/db.go:1001 +0x1b5

2021/10/20 21:12:19 [FATAL] etcd exited

Any help resolving this issue is highly appreciated…

2

Anybody… had any leads on this issue ?

3

There is not enough information, what version of Rancher are you running, is it Single Docker install or HA, what happened before it was not coming up?

4

This looks like a corrupted etcd.
If this node is part of a cluster, I would recommend removing it, cleaning it, and rejoining it to the cluster.
If this is a single node cluster, I would recommend restoring from backup.

5

We are using Rancher v2.5.8, It’s single node install. Rancher UI is running as docker container and container exited after that manually when we are trying to bring it up… it was not coming up and erroring out as mentioned above.