in reference of this post [Rancher Server eat all disk Space]
I purged mysql but it still takes a lot of space and made sure expire_logs_days is set to 2 days.
it was an upgrade
At some point this situation will become problematic, Because it keep growing, Anyone experiencing this ?
there’s not such things as “RTFM”
http://docs.rancher.com/rancher/faqs/troubleshooting/#rancher-server-database-is-growing-too-quickly
after one week we’re still having this issue, mysql is still growing fast
Can you try running this command to see which tables are the largest?
SELECT
ROUND((data_length+index_length)/power(1024,2)) AS tablesize_mb,
table_name
FROM information_schema.tables
WHERE table_schema='cattle'
ORDER BY tablesize_mb DESC;
SELECT ROUND((data_length+index_length)/power(1024,2)) AS tablesize_mb,table_name FROM information_schema.tables WHERE table_schema=‘cattle’ ORDER BY tablesize_mb DESC;
±-------------±----------------------------------------------+
| tablesize_mb | table_name |
±-------------±----------------------------------------------+
| 7248 | process_execution |
| 852 | instance_label_map |
| 659 | instance |
| 302 | label |
| 209 | image |
| 157 | volume |
| 143 | nic |
| 142 | config_item_status |
| 113 | service_expose_map |
| 97 | port |
| 79 | instance_link |
| 44 | agent |
| 40 | credential |
| 27 | process_instance |
| 25 | account |
| 12 | task_instance |
| 7 | load_balancer_target |
| 4 | audit_log |
| 1 | external_handler_external_handler_process_map |
| 0 | ip_association |
| 0 | physical_host |
| 0 | external_handler_process |
| 0 | offering |
| 0 | ip_address_nic_map |
| 0 | task |
| 0 | external_handler |
| 0 | subnet |
| 0 | ip_address |
| 0 | DATABASECHANGELOGLOCK |
| 0 | subnet_vnet_map |
| 0 | network_service_provider_instance_map |
| 0 | external_event |
| 0 | storage_pool_host_map |
| 0 | DATABASECHANGELOG |
| 0 | storage_pool |
| 0 | network_service_provider |
| 0 | environment |
| 0 | snapshot_storage_pool_map |
| 0 | instance_host_map |
| 0 | network_service |
| 0 | dynamic_schema |
| 0 | snapshot |
| 0 | credential_instance_map |
| 0 | network |
| 0 | data |
| 0 | mount |
| 0 | setting |
| 0 | image_storage_pool_map |
| 0 | container_event |
| 0 | load_balancer_listener |
| 0 | host_label_map |
| 0 | service_event |
| 0 | host_vnet_map |
| 0 | load_balancer_host_map |
| 0 | host_ip_address_map |
| 0 | service_consume_map |
| 0 | config_item |
| 0 | host |
| 0 | service |
| 0 | cluster_host_map |
| 0 | zone |
| 0 | load_balancer_config_listener_map |
| 0 | healthcheck_instance_host_map |
| 0 | resource_pool |
| 0 | load_balancer_certificate_map |
| 0 | certificate |
| 0 | volume_storage_pool_map |
| 0 | load_balancer_config |
| 0 | project_member |
| 0 | load_balancer |
| 0 | auth_token |
| 0 | healthcheck_instance |
| 0 | vnet |
| 0 | global_load_balancer |
| 0 | ip_pool |
| 0 | agent_group |
| 0 | user_preference |
| 0 | generic_object |
±-------------±----------------------------------------------+
so what do you think ?
We added some cleanup for the DB awhile ago, but I was just confirming which tables were being cleaned up. It looks like process_execution is not currently in the list and already a known bug.
Sorry, hit enter accidentally. Here’s the bug:
Also, this will end up growing if you have a service reconciliation issue (meaning you have a service that failed and is continually trying to go).
It does not seems there’s a service who keeps restarting, is there a way to chase it through few hosts ?
Is there a way to track a restarting service through several host ? because I can’t see any flapping services
Once it is found is there a way to recover the space ?
You can check the process tab on the Admin to see if anything is restarting there.
In the most recent versions of Rancher (v0.59.0+), the process_execution table should be automatically cleaned up.
once we tracked and fixed all the restarting/unhealthy services, put shorter process_instance , optimized the tables like proposed here : https://github.com/rancher/rancher/issues/3501, we’re still getting short on space, If you have other solutions or work around it’s welcome
Hi,
Please, make sure you do a backup before proceeding.
I believe your issue could be related to this: https://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/ check the size of your database files.
In relation to logs, rotate them or simply send the log to a volatile output. Also you can use data volumes, it should not consume more space than it would normally do outside a container.
The alternatively would be maybe have your database in a virtual machine outside of your Rancher environment, with master-to-master replication for better performance/reliability.