How to collect multi-line logs like Java stack trace logs and ship it to ElasticSearch?

Rancher
1

Right now all the logs that ship to Elasticsearch are single line,it’s hard to view stack trace logs in Kibana.
Is there any way to collect multi-line logs on Rancher?

2

At work we use a logging formatter that ouputs stacktraces on a single line with backslash n for line separation

3

HI, i cant add “parser” into the “edit as a form”

<parse>
  @type multiline
  format_firstline /\d{4}-\d{1,2}-\d{1,2}/
  format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
</parse>

How to configure rancher-logging to send correct java multi line format to elasticsearch?
Need help! Thank you so much!

1 Like
4

The way we fixed this with fluentd native rke2 rancher 2.8.8:

spec:
  filters:
    - concat:
        flush_interval: 1
        key: message
        multiline_start_regexp: ^(\d{4}-\d{1,2}-\d{1,2}T\d{1,2}:\d{1,2}:\d{1,2}.\d{0,3})
        stream_identity_key: kubernetes.pod_name
5

Yes, you can collect multi-line logs in Rancher by configuring your logging driver to handle multi-line patterns. For Elasticsearch, use the grok or logstash format to parse multi-line stack traces. You can adjust the log collection settings in your logging configuration to ensure multi-line events are handled correctly, typically by defining a regex pattern for stack traces.