Built-in MySQL InnoDB file out of control growth on Rancher Server

Archive Rancher 1.x
1

Hi,

Has anybody experiences an issue with the internal MySQL InnoDB database? For some reason it appears that at one point the size blew up to get the main innodb fle over 12Gb, and with journaling logs under CoreOS it ends up filling up my drive and causing the server to crash :frowning:

I solved the issue by exposing MySQL on port 3306 on the rancher/server, backing up the database, and migrating it to a dedicated MySQL server (Google SQL), and everything is behaving much better now. UI is a lot faster, and the actual size of the DB is only ~600Mo.

The MysQL dump itself was only around 300Mo, so how did it blow up to use a 12Gb InnoDB file is beyond me??? Since it is not possible to shrink InnoDB files in MySQL (as far as I know) it is a real problem that can easily cause you to have to rebuild your controller.

Looking for some feedback/comments to see if anybody experienced anything like this or not.

Thank you!

2

Rancher can create lots of rows in a couple tables in a few situations… Periodic cleanup of these was added in 0.35, once a day by default. MySQL then decides to mark pages of the ibdata file reusable whenever it feels like, and then as you said you can’t shrink the files once they grow.

#2087 should go into the next release to make mysql use one file per table by default, which would make it easier to clean up when something like this happens.