What is the recommended way to restart a service like Wordpress after its mysql container moves?
I followed the Rancher Quick Start (2 WP, 1 mysql, 1 load balancer).
As a test, I forced the mysql container to move (& change IP) by removing it from the service.
Then, the mysql container was restarted on another host automatically (sounds good so far).
I confirmed that WP was configured to use hostname=‘mysql’ and not a hard-coded IP address (good).
At this point, mysql is operational again, but WP (apache2) cannot load the home page: Displays “Error establishing a database connection”.
To restore service, I had to manually restart apache2 inside the containers, then all is restored again.
So, what should be done to enable full, automatic restoration of the stack?
Can the WP containers be restarted automatically if their mysql database moves?
The general approach is to make applications resilient to changes. So if your mysql moves, your application (wordpress) should just handle it.
If you have an application that cannot be fixed like this (e.g. Apache2/wordpress), perhaps you can build a small ‘sidekick’ container that watches the dependency (mysql) and restarts the main container if something changes.
Of course, for this, we’d need to know how to trigger a reload of another container from within a different one, which, at this point, I don’t know how to do
Upayavira, thank you for the post.
Yes, it seems in the brave-new world of containers, our services need meta info from orchestration like Rancher, or Rancher needs to have rules to detect and respond to service interruptions caused by Rancher (moving containers).
For example, if apache2 discovers it cannot reach its mysql, should it assume this is because the mysql container moved? Probably not, because mysql could be unreachable for a variety of other “traditional” reasons.
So, should apache2 check with Rancher to see if the mysql container moved? I say no, because the thesis of containers is they don’t have to know they are being controlled by an outside system like Rancher.
So it seems we need the capability in Rancher to restart a container if certain conditions are detected after a Rancher event (like container-moved) happened.
Even this simple Quick-start scenario demonstrates that you cannot just move backend containers and assume the front-end containers will be OK. Even if IP reachability is preserved.
Having said that - Rancher provides service discovery via DNS. So, if you have cross-host networking working, and link your mysql server to your apache one, you can refer to mysql via a hostname, and, depending upon the Rancher DNS TTL, Apache should survive a move of a mysql underneath it because DNS adjusts accordingly.
Yeah, that’s what I too thought should happen. But it doesn’t, at least with the Rancher Quick Start example/tutorial.
As mentioned in the first post of this thread:
“I confirmed that WP was configured to use hostname=‘mysql’ and not a hard-coded IP address (good).”
