Log-aggregator pods crash during install

Rancher 2.x
1

Hi there,

I’m on Rancher 2.0.8 and managing an imported GKE Cluster; I enabled log collection and Rancher createsd the necessary fluentd and log-aggregator pods in cattle-logging. However, the log-aggregator pods immediately crash during initialization with the following reason:

failed to start container "2840026220a08b2b9a9b083ead0f5366be5df30d8d4fca43343fb3a2f183c22d": 
Error response from daemon: error while creating mount source path '/usr/libexec/kubernetes/kubelet-plugins/volume/exec': 
mkdir /usr/libexec/kubernetes: read-only file system: 
RunContainerError

Has anyone seen this before?

Is log collection supposed to work against GKE clusters? (There’s stackdriver logging, that’s why I ask)

Thank you!

Christian Frank
Clouds Sky (www.cloudssky.com)

PS: Here’s the corresponding yaml, as created by Rancher and seen in the GCloud console:

apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "1"
  creationTimestamp: null
  generation: 1
  labels:
    k8s-app: log-aggregator
  name: log-aggregator
  selfLink: /apis/apps/v1beta2/namespaces/cattle-logging/daemonsets/log-aggregator
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: log-aggregator
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: log-aggregator
      name: log-aggregator
      namespace: cattle-logging
    spec:
      containers:
      - image: rancher/log-aggregator:v0.1.3
        imagePullPolicy: Always
        name: log-aggregator
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /flexmnt
          name: flexvolume-driver
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: log-aggregator
      serviceAccountName: log-aggregator
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
          type: ""
        name: flexvolume-driver
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 0
  desiredNumberScheduled: 0
  numberMisscheduled: 0
  numberReady: 0
2

Hi there,
I have exactly the same issue.
Did you find a fix ?

3

Hi Etienne,

yes - the fix is described in Issue 14425 (https://github.com/rancher/rancher/issues/14425)

I haven’t seen it occur in a while though, I thought it was fixed in a recent release.

Which version of Rancher are you using?

Cheers: Chris