I’m trying to build a “multi-tenancy” RKE cluster with Existing Custom Nodes from Rancher 2.5.3. In the cluster I want to dedicate specific worker nodes to different customers using a specific name-space for each customer and NodeSelectors to define which worker nodes each name-space should use.
To prohibit users from violating the policy I’ve managed to enable PodNodeSelector by adding the following lines to the cluster YAML config:
kube-api: extra_args: admission-control-config-file: /etc/kubernetes/adm-control/admission-control.yaml enable-admission-plugins: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodNodeSelector extra_binds: - '/etc/kubernetes/adm-control:/etc/kubernetes/adm-control'
And created the following files on each ctrl-plane node:
$ sudo cat /etc/kubernetes/adm-control/admission-control.yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: PodNodeSelector path: podnodeselector.yaml $ sudo cat /etc/kubernetes/adm-control/podnodeselector.yaml podNodeSelectorPluginConfig: clusterDefaultNodeSelector: "cust=none" ns-cust1: "cust=cust1" ns-cust2: "cust=cust2"
I could then label my hosts and annotate my name-spaces accordingly to make each customers pods scheduled to the correct hosts.
Is there a better way to accomplish the same thing, or an easier way to configure it with Rancher? Especially the need to need to create local config-files on each master node and bind into the kube-apiserver service is really ugly to maintain. Any way to “inject” those config files through the UI or the cluster YAML config instead?