I have seen this questions over and over in different posts left unanswered…here I try again.
What is the maximum number of hosts a standard single node Rancher installation can support? (With support I mean user does not experience slowness in the rancher UI nor via the API.)
Up to how many containers is Rancher Cattle able to support?
we have optimized already the rancher setup by using an external mysql with innodb_buffer_pool_size=2GB see full compose file
we have around
~70 hosts
~10 environments
~ 700 containers
it runs ok…but at times the Rancher server is very busy, CPU goes to the roof for minutes …until it comes back to normal…during this time rancher UI is not responsive.
I have 4 environments (Prod, QA, Dev, and Ops) running across 46 hosts in AWS ranging in size from t2.large to m3.2xlarge instances. I will run anywhere from 1000 - 2500 containers on Cattle.
The bottleneck right now seems to be the rancher server CPU. We’ve found we have to roll out services slowly so we kill the server with all the traffic. You also have to be careful when viewing the infrastructure page or how many tabs open you have with Rancher or the server CPU will spike.
I’d love to see some switches to dial down the responsiveness of the Rancher server so it doesn’t make so many requests.
we ran the mysqltuner.pl on Rancher server’s external mysql. Besides security, these were the recommendations:
General recommendations:
Enable the slow query log to troubleshoot bad queries
Reduce or eliminate unclosed connections and network issues
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
Beware that open_files_limit (1048576) variable
should be greater than table_open_cache ( 400)
Variables to adjust:
query_cache_type (=0)
thread_cache_size (> 8)
table_open_cache (> 400)
innodb_buffer_pool_size (>= 11G) if possible.
innodb_buffer_pool_instances(=2)
Basically, we should have innodb_buffer_pool_instances set to match the number of CPU cores and the VM needs more RAM.
Currently we run on a VM with 4cpu and 8GB RAM. It cannot handle to manage Rancher with ~70 hosts and 700 containers. It is often in high CPU and swapping.
We will move to a larger VM with at least 16 GB RAM which will also have ~ 8 cpus. At least 11 GB ram will be given to innodb_buffer_pool_size and we will also set innodb_buffer_pool_instances to match the number of cpus.
We have scaled vertically the host from 8GB to 16GB and increased the innodb_buffer_pool_size to 11GB and finally Rancher performance is fine. We don’t see anymore CPU spikes since there is enough RAM for both rancher server and the mysql db.
See attached image from our Check_mk monitoring system.