Centralized Log set up for Windows Containers

1

Hi all,

I know this may have been asked earlier, but didnt found any valid solutions.
I am just trying to get some docker containers to production and using Rancher for orchestration.

I am now involved to retrieve application logs from all containers ( 3-4) into a central log repo.

My application logs are stored in a text file and use Log4j.
Any suggestions on the best practice to implement segregation of logs?
I would also be interested to route Rancher logs to the central log repo.

Thanks ,
Vishnu

2

It really depends on whether you have any centralised logging and alerting software that you need to comply with as part of your operational run support (i.e who will be notified if something bad happens, how will that notification be ‘triaged’ from the information available, who will a problem be escalated to, and so on …).

At the most simple level you can configure the docker daemon with a specific logging driver (if there’s one that meets your need). For example, if you use Splunk, it’s easy to configure docker to emit log streams thru that driver which you then hook up either to an event forwarder or file watcher. Rancher is also just a bunch of docker containers so Rancher logs can go the same route. One area that isn’t covered AFAIK is the Rancher audit logs which might be important to your security folk, but in that case you can call the Rancher API periodically and suck those out and forward them on (simplifying here for brevity).

If you want to do more than just output to sys-out, then your app will likely need to include code to push events to a defined logging endpoint. In your case you are sending to a file, so any logging tool that can watch that directory would work, although tbh I’m not a great fan of file based logging especially in a world where statelessness is desirable and you clearly can’t be certain where you app is running and for how long (everything is ephemeral in the docker world). It also tends to introduce a security vulnerability. Better to use a secure network endpoint imho.

3

Thanks @Fraser_Goffin
Apologies for later reply.I got lost in several tasks and missed this.

  1. To start with yes sysout is fine. Will explore Splunk .

  2. I was thinking I can forward Rancher audit logs directly. But after you mentioned we need Rancher API’s to forward those logs, I am wondering is there any utility written already for that.
    Else is there a link for me to get started with. I did a quick search and couldn’t find
    Thanks,
    Vishnu

4

Ranchers audit logs aren’t emitted in the same way as other parts of the platform are. There’s no out of the box solution to that but wrapping the API call and writing some simple logic to parse the results is pretty easy. Stick that in a container and have it call the API periodically and pass the results to your logging app. We did this a while back and it took one of our devs about a day. In our case we pass the results to an AWS Cloudwatch log group which in our case is monitored by an upstream process. One thing to watch out for is ensuring that you don’t get duplicates and more importantly don’t miss any log events (dupes may not matter as much). This can happen if you mess up the filter logic and the periodicity with which you call the API. If you look at the fields that you get back it’s straightforward to create a filter to make sure that doesn’t happen (I don’t have the code to hand or I’d tell you what field we used, but it will be obvious).

5

Thanks a lot @Fraser_Goffin

I will explore that. I got the link Rancher API