Hello. I have horrible issue with my rancher server after VM rebooting.
Here is my logs. If anyone can help me with something – please help
docker ps:
`CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES`
`6f7e7e5a0489 rancher/rancher:v2.2.10 "entrypoint.sh" 3 minutes ago Restarting (2) 27 seconds ago tender_rhodes`
`291528da66df rancher/coreos-etcd:v3.3.10-rancher1 "/usr/local/bin/etcd…" 17 minutes ago Up 17 minutes etcd`
`aaf2e2cf87dd rancher/rancher-agent:v2.3.2 "run.sh --server htt…" 35 minutes ago Restarting (1) 18 seconds ago frosty_lichterman`
`20955501300a rancher/rancher-agent:v2.3.2 "run.sh --server htt…" About an hour ago Restarting (1) 17 seconds ago reverent_allen`
`073dbc9fc504 rancher/rancher-agent:v2.3.2 "run.sh --server htt…" About an hour ago Restarting (1) 17 seconds ago upbeat_hermann`
`42887c6e8542 rancher/hyperkube:v1.15.5-rancher1 "/opt/rke-tools/entr…" 4 months ago Up 35 minutes kube-proxy`
`4837fcd678c5 rancher/hyperkube:v1.15.5-rancher1 "/opt/rke-tools/entr…" 4 months ago Up 35 minutes kubelet`
`16ff09d1f550 rancher/hyperkube:v1.15.5-rancher1 "/opt/rke-tools/entr…" 4 months ago Up 18 minutes kube-scheduler`
`becfe8905349 rancher/hyperkube:v1.15.5-rancher1 "/opt/rke-tools/entr…" 4 months ago Up 18 minutes kube-controller-manager`
`93d149c3d84b rancher/hyperkube:v1.15.5-rancher1 "/opt/rke-tools/entr…" 4 months ago Up 35 minutes kube-apiserver`
docker logs:
2020/03/17 10:10:48 [INFO] Rancher version v2.2.10 is starting
2020/03/17 10:10:48 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false KubeConfig: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false NoCACerts:false ListenConfig:<nil> AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0}
2020/03/17 10:10:48 [INFO] Listening on /tmp/log.sock
2020/03/17 10:10:48 [INFO] Running etcd --data-dir=management-state/etcd
2020-03-17 10:10:48.805711 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2020-03-17 10:10:48.805766 I | etcdmain: etcd Version: 3.2.13
2020-03-17 10:10:48.805800 I | etcdmain: Git SHA: Not provided (use ./build instead of go build)
2020-03-17 10:10:48.805817 I | etcdmain: Go Version: go1.12.10
2020-03-17 10:10:48.805832 I | etcdmain: Go OS/Arch: linux/amd64
2020-03-17 10:10:48.805852 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
2020-03-17 10:10:48.805945 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2020-03-17 10:10:48.806236 I | embed: listening for peers on http://localhost:2380
2020-03-17 10:10:48.806430 I | embed: listening for client requests on localhost:2379
2020-03-17 10:10:48.815126 I | etcdserver: recovered store from snapshot at index 24800248
2020-03-17 10:10:48.822264 C | etcdserver: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
panic: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1dfb3de]
goroutine 168 [running]:
github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdserver.NewServer.func1(0xc00162a690, 0xc00162a380)
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdserver/server.go:284 +0x3e
panic(0x3459940, 0xc0011aa050)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/rancher/rancher/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc00000c400, 0x3e0367f, 0x2a, 0xc00162a450, 0x1, 0x1)
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x135
github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc00048d980, 0xc00048d980, 0x47343a0, 0xc00123e030)
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdserver/server.go:379 +0x2903
github.com/rancher/rancher/vendor/github.com/coreos/etcd/embed.StartEtcd(0xc00104ca80, 0xc001516000, 0x0, 0x0)
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/etcd/embed/etcd.go:157 +0x6dc
github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc00104ca80, 0x3d81b85, 0x6, 0x1, 0xc001513240)
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdmain/etcd.go:186 +0x6a
github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdmain/etcd.go:103 +0x13fb
github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdmain.Main()
/go/src/github.com/rancher/rancher/vendor/github.com/coreos/etcd/etcdmain/main.go:39 +0x118
github.com/rancher/rancher/vendor/github.com/rancher/norman/pkg/kwrapper/etcd.runEtcd(0x47d19a0, 0xc0014aa740, 0xc000a00fe0, 0x2, 0x2)
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/pkg/kwrapper/etcd/etcd.go:68 +0xfe
created by github.com/rancher/rancher/vendor/github.com/rancher/norman/pkg/kwrapper/etcd.RunETCD
/go/src/github.com/rancher/rancher/vendor/github.com/rancher/norman/pkg/kwrapper/etcd/etcd.go:24 +0x187