We have had some problems with containers using EBS-backed volumes. It seems to me that there’s no way to force rancher to “stop trying” in case of volume mount errors or similar - scaling down the service to 0 has no effect, everything is still stuck in “starting” state - even if I go to the aws console and force detach the volume.
For EBS-backed volumes to be production ready, we need more options in terms of breakglass functionality in rancher, such as
- Force stop container
- Force check if the volume is actually mounted
- Force detach volume
We’ve had multiple situations where the only thing we could do was to bring down the entire rancher environment, detach all volumes in aws and then start it up again. “not fun” in dev/test, and completely unacceptable in prod.