AWS host comes up and registers with docker, but when I add my container to it, this message is presented:
Error (Couldn’t bring up network: failed to find plugin “rancher-bridge” in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin])
I have the following ingress ports open to my host:
500 UDP 0.0.0.0/0
4500 UDP 0.0.0.0/0
Per these instructions
Thanks for any help you can provide.
I had this error repeatedly. This seems to be a bug in Rancher Server 1.5.3. Try getting:
rancher/server:v1.5.2 instead.
It solved my issue.
There’s no way that’s anything more than coincidence. 1.5.3 contains only a security fix in an unrelated service vs 1.5.2.
All the same, I won’t move past 1.5.2 based upon my experience in the Rancher Environment. The issue was present at try.rancher.com with 1.5.3 and in my own hosted environment with 1.5.3 and only resolved itself when downgrading to 1.5.2.
Did you do anything else besides downgrading? I went back to 1.5.2 but my ipsec and healthcheck services are still failing with the same error seen in the logs.
I’d recommend evacuating the hosts of their running containers so they get the appropriate infrastructure container versions:
Same error. I even deleted all containers and images to make sure. Evacuate seems to be removing all the user stacks and not the infrastructure ones making no change on the ipsec and healthcheck containers. When I upgraded to 1.5.3 I also upgraded the network stack and ipsec. Looking at them now, I can’t find an earlier version to downgrade to.
ok, not sure if this will help, but I’m running:
My Rancher server is an AWS m4.large instance connected to an AWS RDS MySQL database. I did find things to be really flaky if the database was underpowered.
I have the same versions. Something else must be the cause then. I’ll wait for a response on the github issue I created. They’re slow to respond though.
I’m running 1.12.6 docker.
@wmbutler Do you still have a setup in this state? If you look through the logs of network-manager container, do you see any errors there?
I’m running v1.5.5 under a test cluster right now. So far, so good.
I only experienced errors under 1.5.3. I’ve now skipped to to 1.5.5 and I’ll let you know if I run into anything new.
Same errors with 1.5.5. Tried up to 10 hosts then I gave up.
Same issue
nginxFS (Expected state running but got error: Couldn't bring up network: failed to find plugin "rancher-bridge" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin])
rancher latest stable (1.5.5)
Same issue for me. Running 1.5.5. I’ve been running 3 hosts without issue for weeks. Then I decided to take down one host and add another:
This message repeats contantly @leodotcloud:
4/18/2017 1:30:21 PMtime="2017-04-18T17:30:21Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:21 PMtime="2017-04-18T17:30:21Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:21 PMtime="2017-04-18T17:30:21Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:21 PMtime="2017-04-18T17:30:21Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="Evaluating state from retry" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 count=8
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="CNI up" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 networkMode=ipsec
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:22 PMtime="2017-04-18T17:30:22Z" level=info msg="Setting up resolv.conf for ContainerId [08239d69820cd16bcf077ce8c255573ef2f12b1c48f23721de0cc677cacd8d48]"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:23 PMtime="2017-04-18T17:30:23Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=info msg="Evaluating state from retry" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 count=9
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=info msg="CNI up" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 networkMode=ipsec
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=info msg="Setting up binaries for: map[rancher-bridge:9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302]"
4/18/2017 1:30:24 PMtime="2017-04-18T17:30:24Z" level=error msg="Failed to apply cni conf: Error: No such container: 9038e3e361bcdc5c98cea8f12e38435365c7624da7c28018e6c2300f9770d302"
4/18/2017 1:30:25 PMtime="2017-04-18T17:30:25Z" level=info msg="Setting up resolv.conf for ContainerId [80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52]"
4/18/2017 1:30:25 PMtime="2017-04-18T17:30:25Z" level=info msg="CNI up" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 networkMode=ipsec
4/18/2017 1:30:25 PMtime="2017-04-18T17:30:25Z" level=error msg="Failed to evaluate network state for 80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:25 PMtime="2017-04-18T17:30:25Z" level=error msg="Error processing event &docker.APIEvents{Action:\"start\", Type:\"container\", Actor:docker.APIActor{ID:\"80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52\", Attributes:map[string]string{\"io.rancher.service.hash\":\"43f1a66c7aefadaad1fd69a3af2e6ad50eecd456\", \"io.rancher.service.requested.host.id\":\"16\", \"image\":\"rancher/net:holder\", \"io.rancher.container.ip\":\"10.42.248.85/16\", \"io.rancher.container.system\":\"true\", \"io.rancher.container.uuid\":\"4fb236f8-f738-4a8e-80c8-688f41ee933a\", \"io.rancher.service.deployment.unit\":\"8be4320c-66e1-45d0-a509-8af23df52830\", \"name\":\"r-ipsec-ipsec-3-4fb236f8\", \"io.rancher.stack_service.name\":\"ipsec/ipsec\", \"io.rancher.cni.wait\":\"true\", \"io.rancher.container.mac_address\":\"02:84:66:9b:1e:d6\", \"io.rancher.container.name\":\"ipsec-ipsec-3\", \"io.rancher.project_service.name\":\"ipsec/ipsec\", \"io.rancher.scheduler.global\":\"true\", \"io.rancher.sidekicks\":\"router,cni-driver\", \"io.rancher.stack.name\":\"ipsec\", \"io.rancher.cni.link_mtu_overhead\":\"0\", \"io.rancher.cni.network\":\"ipsec\", \"io.rancher.project.name\":\"ipsec\", \"io.rancher.service.launch.config\":\"io.rancher.service.primary.launch.config\"}}, Status:\"start\", ID:\"80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52\", From:\"rancher/net:holder\", Time:1492536625, TimeNano:1492536625104355794}. Error: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:26 PMtime="2017-04-18T17:30:26Z" level=info msg="Evaluating state from retry" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 count=10
4/18/2017 1:30:26 PMtime="2017-04-18T17:30:26Z" level=info msg="CNI up" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 networkMode=ipsec
4/18/2017 1:30:26 PMtime="2017-04-18T17:30:26Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:27 PMtime="2017-04-18T17:30:27Z" level=info msg="Evaluating state from retry" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 count=1
4/18/2017 1:30:27 PMtime="2017-04-18T17:30:27Z" level=info msg="CNI up" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 networkMode=ipsec
4/18/2017 1:30:27 PMtime="2017-04-18T17:30:27Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:28 PMtime="2017-04-18T17:30:28Z" level=info msg="Evaluating state from retry" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 count=11
4/18/2017 1:30:28 PMtime="2017-04-18T17:30:28Z" level=info msg="CNI up" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 networkMode=ipsec
4/18/2017 1:30:28 PMtime="2017-04-18T17:30:28Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:29 PMtime="2017-04-18T17:30:29Z" level=info msg="Evaluating state from retry" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 count=2
4/18/2017 1:30:29 PMtime="2017-04-18T17:30:29Z" level=info msg="CNI up" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 networkMode=ipsec
4/18/2017 1:30:29 PMtime="2017-04-18T17:30:29Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:30 PMtime="2017-04-18T17:30:30Z" level=info msg="Evaluating state from retry" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 count=12
4/18/2017 1:30:30 PMtime="2017-04-18T17:30:30Z" level=info msg="CNI up" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 networkMode=ipsec
4/18/2017 1:30:30 PMtime="2017-04-18T17:30:30Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:31 PMtime="2017-04-18T17:30:31Z" level=info msg="Evaluating state from retry" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 count=3
4/18/2017 1:30:31 PMtime="2017-04-18T17:30:31Z" level=info msg="CNI up" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 networkMode=ipsec
4/18/2017 1:30:31 PMtime="2017-04-18T17:30:31Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:32 PMtime="2017-04-18T17:30:32Z" level=info msg="Evaluating state from retry" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 count=13
4/18/2017 1:30:32 PMtime="2017-04-18T17:30:32Z" level=info msg="CNI up" cid=f9fb6ae8b284235402a93cb779a3c24919ea8e208c5df161bc8c40545dc36418 networkMode=ipsec
4/18/2017 1:30:32 PMtime="2017-04-18T17:30:32Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
4/18/2017 1:30:33 PMtime="2017-04-18T17:30:33Z" level=info msg="Evaluating state from retry" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 count=4
4/18/2017 1:30:33 PMtime="2017-04-18T17:30:33Z" level=info msg="CNI up" cid=80b0f50ae6230829aafdb0b3b03d6a7074eefee97dadd51c2c812d047716ea52 networkMode=ipsec
4/18/2017 1:30:33 PMtime="2017-04-18T17:30:33Z" level=error msg="Failed to evaluate networking: failed to find plugin \"rancher-bridge\" in path [/opt/cni/bin /var/lib/cni/bin /usr/local/sbin /usr/sbin /sbin /usr/local/bin /usr/bin /bin]"
Seems like whatever is responsible for setting up rancher-bridge, isn’t succeeding.
OS (xenial):
Linux cavo-elasticsearch 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x8
6_64 GNU/Linux
root@cavo-elasticsearch:/usr/local/bin# docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.2
Git commit: 78d1802
Built: Tue Jan 31 23:35:14 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.2
Git commit: 78d1802
Built: Tue Jan 31 23:35:14 2017
OS/Arch: linux/amd64
Downgrading to Rancher Server v1.5.2 resolved the issue once again. So, it seems that if I try to add servers under Rancher > v1.5.2 I consistently get these messages. If I downgrade and then upgrade again, everything works fine.