RKE addons not running

Rancher
1

I’ve setup an RKE template that adds some addons to create some secrets and to start an external cloud-controller-manager. Upon creating a new cluster using this template I can’t see that the secrets are created. Anyone have an example of a working RKE template that demonstrates working addons?

2

We use the rancher2 Terraform provider to create clusters which includes passing an rke_config section to set up add-ons as well as ingress, private_registries and services (etcd, kube_api, kube_controller, kubelet and scheduler). Here is a snippet …

resource "rancher2_cluster" "rancher_cluster" {
    description = "${var.rancher_cluster_name} Rancher Cluster"
    name = "${var.rancher_cluster_name}"
    enable_cluster_alerting = "${var.rancher_cluster_enable_cluster_alerting}"
    enable_cluster_monitoring = "${var.rancher_cluster_enable_cluster_monitoring}"
    cluster_monitoring_input {
        answers = {
            "exporter-node.resources.limits.memory" = "${var.monitoring_exporter_node_memory_limits}"
            "operator.resources.limits.memory" = "${var.monitoring_operator_memory_limits}"
            "prometheus.resources.core.limits.memory" = "${var.monitoring_prometheus_core_memory_limits}"
            "prometheus.resources.core.requests.memory" = "${var.monitoring_prometheus_core_memory_requests}"
        }
    }
    
    rke_config {
        addons = "${file("${path.root}/templates/psp/pod-security-policies-default.yaml")}${file("${path.module}/templates/application-cluster/pod-security-policies-restricted.yaml")}${file("${path.root}/templates/psp/010-pod-security-policy-k8sauth.yaml")}${file("${path.root}/templates/psp/020-pod-security-policy-monitoring.yaml")}"
        ...

In this case there are just PSP templates that are referenced, but it may be of some use (at least to demonstrate that add-ons can be successfully configured). The PSP templates are just regular YAML:

 apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: default-psp-role
  namespace: ingress-nginx
rules:
- apiGroups:
  - extensions
  resourceNames:
  - default-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-psp-rolebinding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: default-psp-role
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated
---
apiVersion: v1
kind: Namespace
metadata:
  name: cattle-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: default-psp-role
  namespace: cattle-system
rules:
- apiGroups:
  - extensions
  resourceNames:
  - default-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
...

The equivalent in a static cluster.yml file for a Rancher management cluster (in HA mode) would be something like:

nodes:
- address: <dns-01>
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: k8s-ha-shared-01
  user: ec2-user
  ssh_key_path: ~/.ssh/id_rke
- address: <dns-02>
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: k8s-ha-shared-02
  user: ec2-user
  ssh_key_path: ~/.ssh/id_rke
- address: <dns-03>
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: k8s-ha-shared-03
  user: ec2-user
  ssh_key_path: ~/.ssh/id_rke
services:
  etcd:
    ...
  kube-api:
    ...
  kube-controller:
    ...
  scheduler:
    ....
  kubelet:
    ...
  kubeproxy:
    ...
addons:
addons_include: [
  "rancher_common/terraform/rancher/cluster/templates/ha/pod-security-policies-default.yaml",
  "rancher_common/terraform/rancher/cluster/templates/ha/pod-security-policies-restricted.yaml",
  "rancher_common/terraform/rancher/cluster/templates/ha/020-pod-security-policy-monitoring.yaml"
]
...

HTHs

Fraser