ES Filebeat workload gives 403 forbidden error trying to access kubernetes api (no service account). How to solve using rancher yaml/api only?

Rancher 2.x
1

Hi

My Rancher version is 2.3.5. Se below for workload yaml and container error related to my problem.

I’m trying to install filebeat without using rancher app (helm). I know that the helm chart for filebeat correctly creates a service account to access the kubernetes api, my manual setup don’t do this and therefor i get an error.

I would like to have complete control of the config, so creating the filebeat helm chart through rancher api is not what i want even if this solves my automation “problem”.

My entire deploy routine is based on using the rancher api, so is there any way i can allow filebeat workload access to kubernetes api without manually creating service account in kubectl? More specifically is there any way to make logstash work in rancher by only using workload yaml or rancher api.

Filebeat container error:
2020-02-07T12:19:05.189Z INFO instance/beat.go:606 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]

7.2.2020 13:19:05 2020-02-07T12:19:05.190Z INFO instance/beat.go:614 Beat ID: 6835c395-a511-4b0c-931e-e47d3e6bff7c

7.2.2020 13:19:05 2020-02-07T12:19:05.190Z INFO kubernetes/util.go:86 kubernetes: Using pod name filebeat-zw28r and namespace filebeat to discover kubernetes node

7.2.2020 13:19:05 2020-02-07T12:19:05.194Z ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: kubernetes api: Failure 403 pods "filebeat-zw28r" is forbidden: User "system:serviceaccount:filebeat:default" cannot get resource "pods" in API group "" in the namespace "filebeat"

7.2.2020 13:19:05 2020-02-07T12:19:05.194Z INFO kubernetes/watcher.go:182 kubernetes: Performing a resource sync for *v1.PodList

7.2.2020 13:19:05 2020-02-07T12:19:05.195Z ERROR kubernetes/watcher.go:185 kubernetes: Performing a resource sync err kubernetes api: Failure 403 pods is forbidden: User "system:serviceaccount:filebeat:default" cannot list resource "pods" in API group "" at the cluster scope for *v1.PodList

7.2.2020 13:19:05 2020-02-07T12:19:05.195Z INFO instance/beat.go:366 filebeat stopped.

7.2.2020 13:19:05 2020-02-07T12:19:05.195Z ERROR instance/beat.go:877 Exiting: error initializing processors: kubernetes api: Failure 403 pods is forbidden: User "system:serviceaccount:filebeat:default" cannot list resource "pods" in API group "" at the cluster scope

7.2.2020 13:19:05 Exiting: error initializing processors: kubernetes api: Failure 403 pods is forbidden: User "system:serviceaccount:filebeat:default" cannot list resource "pods" in API group "" at the cluster scope

This is my workload yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  generation: 1
  labels:
    cattle.io/creator: norman
    workload.user.cattle.io/workloadselector: daemonSet-filebeat-filebeat
  name: filebeat
  namespace: filebeat
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: daemonSet-filebeat-filebeat
  template:
    metadata:
      labels:
        workload.user.cattle.io/workloadselector: daemonSet-filebeat-filebeat
    spec:
      containers:
      - args:
        - -e
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: filebeat
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.serviceAccountName
        image: docker.elastic.co/beats/filebeat:7.2.0
        imagePullPolicy: IfNotPresent
        name: filebeat
        resources: {}
        securityContext:
          allowPrivilegeEscalation: true
          capabilities: {}
          privileged: true
          readOnlyRootFilesystem: false
          runAsNonRoot: false
          runAsUser: 0
        stdin: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        tty: true
        volumeMounts:
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
        - mountPath: /usr/share/filebeat/data
          name: data
        - mountPath: /usr/share/filebeat/filebeat.yml
          name: filebeat-config
          subPath: filebeat.yml
      dnsConfig: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60
      volumes:
      - hostPath:
          path: /var/log
          type: ""
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: varlibdockercontainers
      - hostPath:
          path: /var/lib/filebeat
          type: ""
        name: data
      - configMap:
          defaultMode: 420
          name: filebeat-yml
          optional: false
        name: filebeat-config
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberMisscheduled: 0
  numberReady: 0
  numberUnavailable: 3
  observedGeneration: 1
  updatedNumberScheduled: 3
2

Solved posting create service account yaml to https://my_clyster/v3/clusters/c-XXXXX?action=importYaml

Workload yaml change
Add: spec.template.spec.serviceAccountName: filebeat

Yaml posted to the rancher api endpoint: action=importYaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: filebeat
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
  
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
1 Like