Unable to remove filesystem because device or resource busy

Rancher 1.x
1

I am using docker version:

% docker version                                                                                                                                                  !10001
Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

when I run the rancher-agent on my hosts I can not remove containers anymore, I always get

Unable to remove filesystem for a01c3c0b5835c423bec069c7c2db81046def0060839add148b738bc2a5da273b: remove /var/lib/docker/containers/a01c3c0b5835c423bec069c7c2db81046def0060839add148b738bc2a5da273b/shm: device or resource busy

as soon as I stop rancher-agent I am able to remove the containers

2

Any ideas on how I could get 1.10.1 on a host to try and reproduce?

Would you be able to try using Docker 1.10.3 and see if you have the same issue? I’ve never faced this issue with Docker 1.10.3 and deleting containers.

Which containers are you trying to delete? Also, what OS are you running?

3

I upgraded to docker 1.10.3 and got the same error.

I am using docker-compose v2 with networks and volumes defined. When I do

docker-compose rm -f```

I get the error.

Running on

```Linux nepal 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux```
4

What is your docker-compose.yml for rancher-agent? This seems like it might be related more to docker-compose as we aren’t doing anything specific that should cause this.

5

No, it does not have anything todo with docker-compose, I get the same error when I try to remove the container created by starting the rancher-agent

Looks like this has something todo with http://stackoverflow.com/questions/30550472/docker-container-with-status-dead-after-consul-healthcheck-runs because the containers I can not remove are in status Dead instead of Exited

6

Hi @tompson , I have been facing this issue and found the root cause as docker mount spaces being watched by rancher cattle watcher and cAdvisor processes. This stops us from removing the container as file system is being used by some other process-

/var/lib/cattle/pyagent/cattle/process_watcher.sh
… /var/lib/cattle/etc/cattle/api.crt …

When you run into this ussue, try to see the mount point of the container using-

find /proc/*/mounts | xargs grep b6a61cae046
/proc/23583/mounts:shm /var/lib/docker/containers/b6a61cae0468880eb07da0bf87323c…/shm tmpfs rw,seclabel,nosuid,nodev,noexec,relatime,size=65536k 0 0
Check the PID of the above process and you can see something like
$ ps -ef |grep 23583
root 23583 23288 0 09:48 ? 00:00:00 host-api -cadvisor-url http://*********** -logtostderr=true -ip 0.0.0.0 -port **** -auth=true -host-uuid 296910d1-50c1-4e85-8648-… -public-key /var/lib/cattle/etc/cattle/api.crt -cattle-url

See Unable to remove filesystem - device or resource busy · Issue #20560 · moby/moby · GitHub
I removed the Rancher agents and that resolved the issue. I am not sure what is eing done by Rancher to resolve this