Multi tenant etcd cluster

I would like to run multiple Kubernetes clusters on bare-metal, but I would hate to waste
3 physical servers per cluster for an etcd cluster. Is it possible to use one etcd cluster for multiple Kubernetes clusters? Eg. creating one etcd cluster with Rancher and then add it to the other Kubernetes clusters with the external etcd option? Maybe I missed something but it is not obvious for me if it is possible or not. If it is possible is there anything agains it? Are there any cons?

THX,
Tom

One etcd: not really no, there is no additional level of partitioning available in etcd to separate the clusters. Also the size of the entire database is quite limited so you wouldn’t want to combine many anyway.

Multiple etcd that happen to be in the same machines and different ports & paths: in principle yes, but this is not a situation we really document/support.

K3s can use an external sqlite/postgres/mysql instead of etcd.

not really no, there is no additional level of partitioning available in etcd to separate the clusters

Are you sure? I wonder because there is a required option in RKE configuration for the path of the
configuration in ETCD. I was now about to do exactly this (two clusters with same etcd cluster) and your answer makes me feel I attempt to do something very unhealthy now. :smile:

For example

...
services:
  etcd:
    path: rancher-cluster
    external_urls:
      - https://10.1.0.10:2379
      - https://10.1.0.11:2379
      - https://10.1.0.12:2379
...

Than in another cluster

...
services:
  etcd:
    path: another-cluster
    external_urls:
      - https://10.1.0.10:2379
      - https://10.1.0.11:2379
      - https://10.1.0.12:2379
...

In ETCD the cluster data are prefixed with the path specified in configuration (ETCD v3.3.8)

...
/rancher-cluster/configmaps/kube-system/rke-coredns-addon
/rancher-cluster/configmaps/kube-system/rke-ingress-controller
/rancher-cluster/configmaps/kube-system/rke-metrics-addon
/rancher-cluster/configmaps/kube-system/rke-network-plugin
...
2 Likes

So, something like @seimic approach is possible? Its very interesting for some projects.