Rancher cattle temporary logs are growing indefinitely

Rancher 1.x
1

We have been running Rancher 1.0.x for a while now in production sine its GA release.

However we have just noticed that Rancher server creates cattle .tmp logs files which seems to be never removed/cleaned up …so they eat up lot of space on the server.

This is the current situation:

we are on rancher 1.0.1. See the current undesired situation with huge .tmp files that are never deleted…I expect rancher to delete them itself. our server crashed because of this.

root@2fbdf877da7a:/var/lib/cattle/logs# ls -lh
total 15G
-rw-r--r--. 1 root root    3616623 Apr 11 02:00 cattle-debug.2016-04-10.log.gz
-rw-r--r--. 1 root root   10080577 Apr 12 02:00 cattle-debug.2016-04-11.log.gz
-rw-r--r--. 1 root root   26416665 Apr 13 02:00 cattle-debug.2016-04-12.log.gz
-rw-r--r--. 1 root root   26188300 Apr 14 02:00 cattle-debug.2016-04-13.log.gz
-rw-r--r--. 1 root root   16296914 Apr 15 02:00 cattle-debug.2016-04-14.log.gz
-rw-r--r--. 1 root root   13780663 Apr 16 02:00 cattle-debug.2016-04-15.log.gz
-rw-r--r--. 1 root root   13442126 Apr 17 02:00 cattle-debug.2016-04-16.log.gz
-rw-r--r--. 1 root root   13371954 Apr 18 02:00 cattle-debug.2016-04-17.log.gz
-rw-r--r--. 1 root root   13785917 Apr 19 02:00 cattle-debug.2016-04-18.log.gz
-rw-r--r--. 1 root root   14068817 Apr 20 02:00 cattle-debug.2016-04-19.log.gz
-rw-r--r--. 1 root root   13769198 Apr 21 02:00 cattle-debug.2016-04-20.log.gz
-rw-r--r--. 1 root root   14020577 Apr 22 02:00 cattle-debug.2016-04-21.log.gz
-rw-r--r--. 1 root root    4521191 Apr 23 02:00 cattle-debug.2016-04-22.log.gz
-rw-r--r--. 1 root root    5256146 Apr 24 02:00 cattle-debug.2016-04-23.log.gz
-rw-r--r--. 1 root root    4349392 Apr 25 02:00 cattle-debug.2016-04-24.log.gz
-rw-r--r--. 1 root root    5170880 Apr 26 02:00 cattle-debug.2016-04-25.log.gz
-rw-r--r--. 1 root root    5229840 Apr 27 02:00 cattle-debug.2016-04-26.log.gz
-rw-r--r--. 1 root root    4426412 Apr 28 02:00 cattle-debug.2016-04-27.log.gz
-rw-r--r--. 1 root root    7373921 Apr 29 02:00 cattle-debug.2016-04-28.log.gz
-rw-r--r--. 1 root root    5240734 Apr 30 02:00 cattle-debug.2016-04-29.log.gz
-rw-r--r--. 1 root root    6186057 May  1 02:00 cattle-debug.2016-04-30.log.gz
-rw-r--r--. 1 root root    5590913 May  2 02:00 cattle-debug.2016-05-01.log.gz
-rw-r--r--. 1 root root    7320029 May  3 02:00 cattle-debug.2016-05-02.log.gz
-rw-r--r--. 1 root root   34909908 May  4 02:00 cattle-debug.2016-05-03.log.gz
-rw-r--r--. 1 root root   46470978 May  5 02:00 cattle-debug.2016-05-04.log.gz
-rw-r--r--. 1 root root   41325638 May  5 11:29 cattle-debug.2016-05-05.log.gz
-rw-r--r--. 1 root root   44818120 May  6 11:02 cattle-debug.2016-05-06.log.gz
-rw-r--r--. 1 root root   64912972 May  7 21:53 cattle-debug.2016-05-07.log.gz
-rw-r--r--. 1 root root   37947657 May  8 08:53 cattle-debug.2016-05-08.log.gz
-rw-r--r--. 1 root root   54382757 May  9 12:30 cattle-debug.2016-05-09.log.gz
-rw-r--r--. 1 root root   22176866 May 10 05:28 cattle-debug.2016-05-10.log.gz
-rw-r--r--. 1 root root  339335267 May 10 10:00 cattle-debug.log
-rw-r--r--. 1 root root 1073742821 May  5 21:36 cattle-debug.log123279789857127.tmp
-rw-r--r--. 1 root root  454023239 May  6 01:59 cattle-debug.log139069242902314.tmp
-rw-r--r--. 1 root root 1073744868 May  6 13:26 cattle-debug.log180252244661795.tmp
-rw-r--r--. 1 root root  770165852 May  7 01:59 cattle-debug.log225467195719777.tmp
-rw-r--r--. 1 root root  223657321 May  8 01:59 cattle-debug.log311865602972644.tmp
-rw-r--r--. 1 root root 1073745429 May  8 11:04 cattle-debug.log344541221251161.tmp
-rw-r--r--. 1 root root 1073743036 May  8 13:15 cattle-debug.log352423666911846.tmp
-rw-r--r--. 1 root root 1073744440 May  8 15:25 cattle-debug.log360202136775544.tmp
-rw-r--r--. 1 root root 1073745901 May  8 17:35 cattle-debug.log367969405601005.tmp
-rw-r--r--. 1 root root  718147786 May  9 01:59 cattle-debug.log398266002053867.tmp
-rw-r--r--. 1 root root 1073746549 May  9 16:53 cattle-debug.log451905266953631.tmp
-rw-r--r--. 1 root root 1073742101 May  9 20:14 cattle-debug.log463910235216070.tmp
-rw-r--r--. 1 root root 1073743960 May  9 23:21 cattle-debug.log475134628399076.tmp
-rw-r--r--. 1 root root  837150726 May 10 01:59 cattle-debug.log484668479728887.tmp
-rw-r--r--. 1 root root 1073744104 May  5 12:11 cattle-debug.log89370040680529.tmp
-rw-r--r--. 1 root root     379182 Apr 22 06:39 cattle-error.2016-04-21.log.gz
-rw-r--r--. 1 root root       7565 Apr 23 03:30 cattle-error.2016-04-22.log.gz
-rw-r--r--. 1 root root      10616 Apr 24 02:53 cattle-error.2016-04-23.log.gz
-rw-r--r--. 1 root root       5876 Apr 25 02:30 cattle-error.2016-04-24.log.gz
-rw-r--r--. 1 root root      13331 Apr 26 04:35 cattle-error.2016-04-25.log.gz
-rw-r--r--. 1 root root      17338 Apr 27 15:21 cattle-error.2016-04-26.log.gz
-rw-r--r--. 1 root root       5355 Apr 28 06:56 cattle-error.2016-04-27.log.gz
-rw-r--r--. 1 root root      55698 Apr 29 06:31 cattle-error.2016-04-28.log.gz
-rw-r--r--. 1 root root       5433 Apr 30 15:32 cattle-error.2016-04-29.log.gz
-rw-r--r--. 1 root root      29225 May  1 03:12 cattle-error.2016-04-30.log.gz
-rw-r--r--. 1 root root      12309 May  2 02:07 cattle-error.2016-05-01.log.gz
-rw-r--r--. 1 root root      41745 May  3 02:10 cattle-error.2016-05-02.log.gz
-rw-r--r--. 1 root root     705821 May  4 02:01 cattle-error.2016-05-03.log.gz
-rw-r--r--. 1 root root     804258 May  5 02:00 cattle-error.2016-05-04.log.gz
-rw-r--r--. 1 root root    3248960 May  6 02:00 cattle-error.2016-05-05.log.gz
-rw-r--r--. 1 root root    5771468 May  7 02:00 cattle-error.2016-05-06.log.gz
-rw-r--r--. 1 root root    4494922 May  8 02:00 cattle-error.2016-05-07.log.gz
-rw-r--r--. 1 root root    2402093 May  9 02:00 cattle-error.2016-05-08.log.gz
-rw-r--r--. 1 root root    3511086 May 10 02:00 cattle-error.2016-05-09.log.gz
-rw-r--r--. 1 root root    9262076 May 11 02:13 cattle-error.2016-05-10.log.gz
-rw-r--r--. 1 root root    7236942 May 12 02:05 cattle-error.2016-05-11.log.gz
-rw-r--r--. 1 root root    6040423 May 13 02:04 cattle-error.2016-05-12.log.gz
-rw-r--r--. 1 root root    7342225 May 14 02:04 cattle-error.2016-05-13.log.gz
-rw-r--r--. 1 root root    8766257 May 15 02:04 cattle-error.2016-05-14.log.gz
-rw-r--r--. 1 root root   10843837 May 16 02:07 cattle-error.2016-05-15.log.gz
-rw-r--r--. 1 root root    9029674 May 17 02:03 cattle-error.2016-05-16.log.gz
-rw-r--r--. 1 root root   15014986 May 18 01:24 cattle-error.2016-05-17.log.gz
-rw-r--r--. 1 root root   10486872 May 19 02:03 cattle-error.2016-05-18.log.gz
-rw-r--r--. 1 root root   14540955 May 20 02:07 cattle-error.2016-05-19.log.gz
-rw-r--r--. 1 root root   10061248 May 21 02:06 cattle-error.2016-05-20.log.gz
-rw-r--r--. 1 root root  257947779 May 21 12:33 cattle-error.log
-rw-r--r--. 1 root root  984259307 May 18 01:57 cattle-error.log456382747586183.tmp

I have also posted a comment here

What shall we do in the mean time to avoid server crashes?
recreate the rancher server container (we have a distinct one for the mysql data) every week? What is Rancher proper planned solution to this?

1 Like
2

Hej Rancher folks!
Do you have any chance to give us some hints here?

We had now to ssh into the rancher server and delete the cattle tmp logs.

But Rancher seems to always create tmp logs there and does not clean up after itself

ls -lha /var/lib/cattle/logs/
…
-rw-r–r--. 1 root root 1.1G May 24 00:30 cattle-error.log969341844228295.tmp
-rw-r–r--. 1 root root 1.1G May 24 00:51 cattle-error.log970591302445187.tmp
-rw-r–r--. 1 root root 50M May 24 01:59 cattle-error.log974721337057276.tmp
…

What is Rancher opinion on this? Do we have to schedule a recreation of container each week?
Or should we deploye by bind mounting on host /var/lib/cattle/logs/ so we can via a cron delete them periodically?
I find this a serious bug for a production running container…

3

Here is something odd. I am on Rancher 1.0.1 and I don’t even have a folder named cattle under /var/lib

So where are my logs going?

4

Rancher seems ignoring us here, probably because it is a corner case for them, under-prioritised.

The funny thing is, that a good container practice is to never store logs inside of containers, they must be placed outside…but rancher server does it and it does it wrongly…so sad that rancher is not doing anything about this.

Right now, to quickly solve this important issue for us, I will re-launch the rancher server with a bind mount option (-v) for the /var/lib/cattle/logs/, so that the logs are placed outside of the container on the host and ops can purge them as desired.

5

Same here, Rancher v1.0.2 and don’t have the path /var/lib/cattle/. Would be good to know where logs go on Rancher server and RancherOS hosts.

6

it is clear now.
In rancher 1.0.2 they disabled the logging


:slight_smile:
that is why. and me being in 1.0.1 was still having issues with the logs.

So case is closed. everybody should upgrade to at least 1.0.2 to fix the logs issues

7

Was this issue resolved in 1.6.5?
The logs are being generated in /var/lib/docker/volumes/c9a2c225f02c376bb0a8ef560d34fc46c5eb9dd22689c3766ee771ccf873c86b/_data/logs

What kind of information you store in debug-cattle.log ?