Hello,
This is not a problem post, but rather I finally managed to configure the fluentd to send logs almost live to the loki log server and want to share it.
The upstream banzaicloud logoperator had a bug with buffer configuration, when it would assume that you are using file buffers and memory buffer configuration could not be applied.
(here is my bug report, where I have described the problem in some details)
So finally now with rancher 2.6.4 we have the patched version with working memory buffers.
My goal was to have minimal delay in the fluentd, so the logs can be “tailed” from the loki server. After disabling the generated buffers config, fluentd still had about 60s delay by using its defaults.
I have tried setting interval to 1s, 10s, tweaking timekey etc to no avail. The delay got bigger, or the logs stopped altogether.
Until I found ( from this article )that I can increase the flush_thread_count (I was sending all the cluster logs, so there were plenty)
Setting flush_thread_count: 8 in the loki output buffer section did the trick and now logs are flushed every second.
I hope someone else can benefit from this too