Updating Rancher Fluentd Logging

Rancher
1

Hi, I was wondering how I can override the log configuration in Rancher.

As I understand, rancher loads fluentd image configuration and modifies it with its own settings. Here is an example of a project.conf running within a fluentd container.

image
image473Ă—948 15.2 KB

What would be the concrete steps I have to take to override this project.conf with a different record_transformer ( to add or remove my own keywords ) or put in a multiline filter stanza?

Lets consider the most basic case, lets say I want to remove the projectID from the configuration. What would I need to do for this to happen in rancher? Do I have modify the base fluentd image? The kubernetes_metadata_filter plugin? Some rancher code?

<filter c-nllvr:p-78zvn.**>
@type record_transformer
enable_ruby true

tag ${tag}
namespace ${record[“kubernetes”][“namespace_name”]}
projectID c-nllvr:p-78zvn <— how I can remove this ?

I assumed I have to edit some base fluentd.conf file somewhere but I wasn’t sure where it was set for rancher. It seems that rancher keeps overriding any fluentd settings you’d throw at it. I checked the rancher source code and found a fluentd project logging template in ./rancher/pkg/controllers/user/logging/generator/project_template.go

@type tail path /var/log/containers/*.log pos_file /fluentd/log/fluentd-project-{{$store.ProjectName}}-logging.pos time_format %Y-%m-%dT%H:%M:%S tag {{$store.ProjectName}}.* format json read_from_head true

<filter {{$store.ProjectName}}.**>
@type kubernetes_metadata
merge_json_log true
preserve_json_log true

<filter {{$store.ProjectName}}.**>
@type record_transformer
enable_ruby true

tag ${tag}
namespace ${record[“kubernetes”][“namespace_name”]}
{{range $k, $val := $store.OutputTags -}}
{{$k}} {{$val}}
{{end -}}
projectID {{$store.ProjectName}}

<filter {{$store.ProjectName}}.**>
@type grep

key namespace
pattern {{$store.GrepNamespace}}

It seems to match perfectly to what I see running in the fluentd containers. But how do (should?) I override that? If I start editing an internal rancher template source code I would have to restart rancher after every filtering change and also have to branch from the main rancher release. There has to be a better (correct) way to edit per-project fluentd configuration without affecting everything else.

Thanks,
Ilya

2 Likes
2

Update with Solution:

Just a note, within rancher the json output of a docker daemon and the built-in fluentd are immutable unless you want to change rancher sourcecode (not recommended).

Solution was to spin up my own fluentd server with this plugin

It will automatically pick up json inputs from rancher and then, using a regex, will combine them into a single message. I’ve tested it and it works. The best thing is you can use the stream_identity_key tag to uniquely identify message sources so you don’t try to combine items from various locations creating a Frankenstein event.

Example I used for testing:

Dockerfile:

FROM fluent/fluentd:v1.6-1
USER root
RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && gem install fluent-plugin-concat \
 && gem install fluent-plugin-splunk-enterprise \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/

Fluentd Conf (sending to splunk HEC in this example)

<source>
  @type  forward
  port  12345
  bind 0.0.0.0
</source>

<filter **>
  @type concat
  key log
  multiline_start_regexp /\w+ \d+.\d+.\d+ \d+:\d+:\d+: /
  stream_identity_key tag
</filter>

<match **>
    @type splunk_hec
    host my_splunk_hec_server
    port 8088
    token MYTOKEN_8675309
    use_ssl true
    default_sourcetype mysoure
    ssl_verify false
    <buffer>
        @type file
        path /var/log/mybuff
        flush_interval 3s
    </buffer>
</match>