CIS Kubernetes Benchmark: Problem with PSP and cis-operator/kube-bench

k3s, k3OS, and k3d
1

Hello,

I’m trying to establish Kubernetes (v1.21) cluster which will pass CIS k8s benchmark. Cluster is based on K3s and I’m using Rancher as UI for easier access. As part of the Rancher 2.6 I also installed CIS operator which has UI where it’s possible to run the scans for checking compatibility of cluster’s config state with expected state from CIS benchmark.

I followed the hardening guide from docs (Rancher Docs: CIS Hardening Guide) and applied PodSecurityPolicy (the same one which is in docs) to the cluster along enabling required admission plugins. This works as expected and limits possibilities of pods. But there is a small problem, because this limits all pods which also applies to the pods created by CIS operator from Rancher. The result of PSP is then limitation of volume/ hostPath usage, which is used for pod containing container with rancher/security-scan image. Here is a snippet of volumes part of yaml config file with hostPaths:

volumes:
  - hostPath:
      path: /var/lib/rancher
      type: ""
    name: rke2-root
  - hostPath:
      path: /etc/cni/net.d
      type: ""
    name: rke2-cni
  - hostPath:
      path: /etc/passwd
      type: ""
    name: etc-passwd
  - hostPath:
      path: /etc/group
      type: ""
    name: etc-group
  - hostPath:
      path: /var/log
      type: ""
    name: var-log
  - hostPath:
      path: /run/log
      type: ""

After I create scan for profile k3s-cis-1.6-hardened , the job starts running and pods for checking the state are being created. Later then the scan fails with the next error:

time="2022-01-28T09:55:46Z" level=info msg="Running Summarizer"
time="2022-01-28T09:55:46Z" level=debug msg="version_mapping: map[rke-1.13:rke-cis-1.4 rke-1.16:rke-cis-1.6-permissive v1.20.5+k3s1:k3s-cis-1.6-permissive rke-1.17:rke-cis-1.6-permissive gke-1.0:gke-1.0 v1.18.10+rke2r1:rke2-cis-1.5-permissive rke-1.18:rke-cis-1.6-permissive rke-1.19:rke-cis-1.6-permissive rke-1.14:rke-cis-1.4 v1.20.5+rke2r1:rke2-cis-1.6-permissive eks-1.0:eks-1.0 rke-1.15:rke-cis-1.5-permissive]"
time="2022-01-28T09:55:46Z" level=info msg="CONFIG: map[rke-1.19:rke-cis-1.6-permissive rke-1.14:rke-cis-1.4 v1.20.5+rke2r1:rke2-cis-1.6-permissive eks-1.0:eks-1.0 rke-1.15:rke-cis-1.5-permissive rke-1.18:rke-cis-1.6-permissive rke-1.16:rke-cis-1.6-permissive v1.20.5+k3s1:k3s-cis-1.6-permissive rke-1.17:rke-cis-1.6-permissive gke-1.0:gke-1.0 v1.18.10+rke2r1:rke2-cis-1.5-permissive rke-1.13:rke-cis-1.4]\n"
time="2022-01-28T09:55:46Z" level=debug msg="filePath: /etc/kube-bench/cfg/k3s-cis-1.6-hardened/controlplane.yaml, controls: &{ID:3 Version:1.6 Text:Control Plane Configuration Type:controlplane Groups:[0xc000093200 0xc000093260] Summary:{Pass:0 Fail:0 Warn:0 Info:0}}"
time="2022-01-28T09:55:46Z" level=debug msg="filePath: /etc/kube-bench/cfg/k3s-cis-1.6-hardened/policies.yaml, controls: &{ID:5 Version:1.6 Text:Kubernetes Policies Type:policies Groups:[0xc0000933e0 0xc000093440 0xc0000934a0 0xc000093500 0xc000093560 0xc0000935c0] Summary:{Pass:0 Fail:0 Warn:0 Info:0}}"
time="2022-01-28T09:55:46Z" level=debug msg="filePath: /etc/kube-bench/cfg/k3s-cis-1.6-hardened/master.yaml, controls: &{ID:1 Version:1.6 Text:Master Node Security Configuration Type:master Groups:[0xc000093740 0xc0000937a0 0xc000093800 0xc000093860] Summary:{Pass:0 Fail:0 Warn:0 Info:0}}"
time="2022-01-28T09:55:46Z" level=debug msg="filePath: /etc/kube-bench/cfg/k3s-cis-1.6-hardened/etcd.yaml, controls: &{ID:2 Version:1.6 Text:Etcd Node Configuration Type:etcd Groups:[0xc0000939e0] Summary:{Pass:0 Fail:0 Warn:0 Info:0}}"
time="2022-01-28T09:55:46Z" level=debug msg="filePath: /etc/kube-bench/cfg/k3s-cis-1.6-hardened/node.yaml, controls: &{ID:4 Version:1.6 Text:Worker Node Security Configuration Type:node Groups:[0xc000093b60 0xc000093bc0] Summary:{Pass:0 Fail:0 Warn:0 Info:0}}"
time="2022-01-28T09:55:46Z" level=debug msg="filePath: /etc/kube-bench/cfg/k3s-cis-1.6-hardened/config.yaml, controls: &{ID: Version: Text: Type: Groups:[] Summary:{Pass:0 Fail:0 Warn:0 Info:0}}"
time="2022-01-28T09:55:46Z" level=debug msg="total groups loaded: 15"
time="2022-01-28T09:55:46Z" level=debug msg="total controls loaded: 122"
time="2022-01-28T09:55:46Z" level=info msg=summarize
time="2022-01-28T09:55:46Z" level=fatal msg="error summarizing: error listing directory: open /tmp/kb-summarizer/input/plugins/rancher-kube-bench/results: no such file or directory"
error running kb-summarizer using override benchmark version

I believe this is connected directly to the hostPath which use is prevented by PSP.

I am not sure how to avoid this error rather then changing security standard from restricted to something else or including hostPath as option on volumes list along others (but still not recommended in official k8s docs). I was looking if it’s possible to create exception for single pod where hostPath could be used, but currently I don’t have any result on that (since PSP is global this is probably not possible at all?).

What would be the best thing to do in that case/how do you bypass this issue?

Any help would be greatly appreciated. Thank you!

(github issue @ https://github.com/k3s-io/k3s/issues/5058)