Using custom Docker log drivers with Rancher

[alecghica]

Right now, with Rancher 1.0, it is possible to use custom Docker log drivers with Rancher like described in this article: http://blog.yjwong.name/2016/03/23/using-custom-docker-log-drivers-with-rancher

The problem is that when using custom log driver, like GELF, the logs from Rancher UI stop working and a error message is displayed:

• Invalid Date Invalid Daterunning logs job: configured logging reader does not support reading

It is possible to make both logs work in the same time?

[johnrengelman]

Somewhat kind of on this topic -
I was hoping to be able to use a log driver and point it at another service that was running in Rancher, but that doesn’t work (probably for obvious reasons). I would be awesome though

[demarant]

there was a similar question in the forum here (still unanswered)

Using log_driver with Gelf instead of logspunk

Hi, I found this post Docker Logging Driver but its closed, so I couldnt continue on there but its a related issue… It seems that using log_driver with Gelf would be a good way to substitute the requirement of Logspout, wouldnt it? Has anyone attempted this? Where are rancher itnernal containers sending their logs? is there a way to configure these to also use a log driver? (maybe a cluster/env wide log driver option?) On a semi-related note, is it possible/planned to be able to change cata…

@alecghica maybe using rancher-compose directly (instead of docker-compose) would make it work?
I am quoting “rancher-compose should already work with it” from @ibuildthecloud

Docker Logging Driver

You’re in luck, we already do! Rancher Labs has contributed a bunch to the log driver (syslog in 1.6 and rollover in 1.8) so we added support for those before docker-compose did. I don’t think there is support in the UI, but rancher-compose should already work with it. Please let me know if it doesn’t work.

hey rancher folks … could you please cast some light on this topic?

[vincent]

I don’t know anything about gelf, but this sounds like the logging driver has no integration to read data back from it. Does docker logs on the host show the same thing? The only info we have to show comes from asking the docker daemon.

[demarant]

@vincent:

the docker logs command on the host is also not able to show the logs.

docker-compose.yml

hello:
  image: hello-world
  log_driver: "gelf"
  log_opt:
    gelf-address: "udp://logcentral.eea.europa.eu:12201"

I run with pure docker-compose up -d and I get the following

output from docker logs

[marinis@dev-fra-02 gelf-test]$ docker logs gelftest_hello_1
"logs" command is supported only for "json-file" and "journald" logging drivers (got: gelf)

output from docker-compose logs

[marinis@dev-fra-02 gelf-test]$ docker-compose logs
Attaching to gelftest_hello_1
hello_1 | Error attaching: configured logging reader does not support reading
gelftest_hello_1 exited with code 0

output from Rancher log console

Invalid Date Invalid Daterunning logs job: configured logging reader does not support reading

We are on CentOS 7 kernel 3.10.0-327.13.1.el7.x86_64
Docker 1.10.3
Rancher 1.0.1

so the question is: How can we centralise logs and at the same time been able to see them in the rancher UI log console? Is it supposed to work somehow?
Any help appriciated

[vincent]

As the error says, the only 2 drivers that implement reading back log entries ( https://github.com/docker/docker/blob/master/daemon/logger/jsonfilelog/read.go ) are docker json and systemd journal. I can’t show you information we can’t get from Docker.

[demarant]

@vincent
thanks, it is clear now the limitation is at the docker daemon level. nothing about Rancher.

So I guess we should probably try journald and from journald to graylog (as this post suggests). Let’s see how that works.

[demarant]

so I tested with journald and it works fine

TL;DR

choosing journald as log plugin for docker will make the logs appearing in all tools (docker logs, docker-compose logs, journalctl on host, syslog on host and be able to forward to graylog2 server)

Details on how to set it up

this is what I did, so if others are looking at the same solution

enabled forward to rsyslog from journald
vi nano /etc/systemd/journald.conf
added the line:
ForwardToSyslog=yes

made syslog forward to external graylog2 server
(or any other centralised log management)
Added the following in /etc/rsyslog.conf

$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
*.* @@logcentral.eea.europa.eu:1514;RSYSLOG_SyslogProtocol23Format

restarted the rsyslog service.

docker-compose example logging via journald
created a sample app with journald logging

hello:
  image: hello-world
  log_driver: "journald"

and when started you can see the logs both with docker-compose logs, docker logs or within Rancher UI.

You can also see them with journalctl CONTAINER_NAME=<container_name>

The only issue I found is that in graylog2 there is no info on from which container the logs comes from. One way to achieve this is to use the tag option as which will add some identification to each log line:

 log_opt:
    tag: "{{.ImageName}}\{{.Name}}\{{.ID}}"

BUT the tag option for journald has been added to docker 1.11, and Rancher only supports docker 1.10.x right now. Of course docker 1.11 will soon be supported looks like
https://github.com/rancher/rancher/issues/4491

[blaggacao]

This feels somewhat “half the way” as it needs special provisioning for the host. By this we turn to configuration management systems again. Schedulers set out to deprecate them.

I’d rather see rancher to allow consuming logs from other services through a configurable plugin based architecture. As docker logging option tag has a default or is otherwise configurable AND always known to rancher, the log identity can be easily detected in most systems.

[kbroughton]

I would like to see the setting available in rancher-compose.yml not just the docker-compose.yml as these responses suggest.

For splunk we have to add 5 lines per container. In a 20 container stack, that’s a lot of hand editing. Allowing over-ride with precedence docker-compose.yml beats rancher-compose.yml beats /etc/defaults/docker would be nice.