Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener

not sure what happened where but I am not able to create a cluster with rke, how can I fix this?

(⎈ |k8s.prime.iad0.netskope.com:default)➜  rancher-npe git:(SYS-1307) ✗ ./rke up --config ./rke-cluster.yml
INFO[0000] Running RKE version: v1.0.4                  
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [dialer] Setup tunnel for host [knode3.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [kmaster3.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [kmaster1.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [kmaster2.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [knode1.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [knode2.prime.iad0.netskope.com] 
INFO[0001] Checking if container [cluster-state-deployer] is running on host [kmaster3.prime.iad0.netskope.com], try #1 
INFO[0001] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster3.prime.iad0.netskope.com] 
INFO[0002] Starting container [cluster-state-deployer] on host [kmaster3.prime.iad0.netskope.com], try #1 
INFO[0004] [state] Successfully started [cluster-state-deployer] container on host [kmaster3.prime.iad0.netskope.com] 
INFO[0004] Checking if container [cluster-state-deployer] is running on host [knode1.prime.iad0.netskope.com], try #1 
INFO[0005] Image [rancher/rke-tools:v0.1.52] exists on host [knode1.prime.iad0.netskope.com] 
INFO[0006] Starting container [cluster-state-deployer] on host [knode1.prime.iad0.netskope.com], try #1 
INFO[0007] [state] Successfully started [cluster-state-deployer] container on host [knode1.prime.iad0.netskope.com] 
INFO[0009] Checking if container [cluster-state-deployer] is running on host [knode2.prime.iad0.netskope.com], try #1 
INFO[0009] Image [rancher/rke-tools:v0.1.52] exists on host [knode2.prime.iad0.netskope.com] 
INFO[0009] Starting container [cluster-state-deployer] on host [knode2.prime.iad0.netskope.com], try #1 
INFO[0010] [state] Successfully started [cluster-state-deployer] container on host [knode2.prime.iad0.netskope.com] 
INFO[0010] Checking if container [cluster-state-deployer] is running on host [knode3.prime.iad0.netskope.com], try #1 
INFO[0011] Image [rancher/rke-tools:v0.1.52] exists on host [knode3.prime.iad0.netskope.com] 
INFO[0012] Starting container [cluster-state-deployer] on host [knode3.prime.iad0.netskope.com], try #1 
INFO[0013] [state] Successfully started [cluster-state-deployer] container on host [knode3.prime.iad0.netskope.com] 
INFO[0014] Checking if container [cluster-state-deployer] is running on host [kmaster1.prime.iad0.netskope.com], try #1 
INFO[0014] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster1.prime.iad0.netskope.com] 
INFO[0015] Starting container [cluster-state-deployer] on host [kmaster1.prime.iad0.netskope.com], try #1 
INFO[0016] [state] Successfully started [cluster-state-deployer] container on host [kmaster1.prime.iad0.netskope.com] 
INFO[0017] Checking if container [cluster-state-deployer] is running on host [kmaster2.prime.iad0.netskope.com], try #1 
INFO[0018] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster2.prime.iad0.netskope.com] 
INFO[0018] Starting container [cluster-state-deployer] on host [kmaster2.prime.iad0.netskope.com], try #1 
INFO[0019] [state] Successfully started [cluster-state-deployer] container on host [kmaster2.prime.iad0.netskope.com] 
INFO[0019] [certificates] Generating CA kubernetes certificates 
INFO[0019] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0020] [certificates] Generating Kubernetes API server certificates 
INFO[0021] [certificates] Generating Service account token key 
INFO[0021] [certificates] Generating Kube Controller certificates 
INFO[0021] [certificates] Generating Kube Scheduler certificates 
INFO[0021] [certificates] Generating Kube Proxy certificates 
INFO[0022] [certificates] Generating Node certificate   
INFO[0022] [certificates] Generating admin certificates and kubeconfig 
INFO[0022] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0022] [certificates] Generating kube-etcd-kmaster1-prime-iad0-netskope-com certificate and key 
INFO[0023] [certificates] Generating kube-etcd-kmaster2-prime-iad0-netskope-com certificate and key 
INFO[0023] [certificates] Generating kube-etcd-kmaster3-prime-iad0-netskope-com certificate and key 
INFO[0024] Successfully Deployed state file at [./rke-cluster.rkestate] 
INFO[0024] Building Kubernetes cluster                  
INFO[0024] [dialer] Setup tunnel for host [kmaster1.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [kmaster2.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [kmaster3.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [knode2.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [knode3.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [knode1.prime.iad0.netskope.com] 
INFO[0025] [network] Deploying port listener containers 
INFO[0025] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster2.prime.iad0.netskope.com] 
INFO[0025] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster3.prime.iad0.netskope.com] 
INFO[0025] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster1.prime.iad0.netskope.com] 
INFO[0025] Starting container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com], try #1 
WARN[0026] Can't start Docker container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (2af2d5c08691e5f1bb1d3a2c69da574ce26dacaad139d0c9ed0e7dc4b7e977ab):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com], try #2 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com], try #1 
WARN[0026] Can't start Docker container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (4afa68ede7789afae27e5b744795c7e65cd5db20cb4e3805fcc2dc4b439138b2):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com], try #3 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com], try #1 
WARN[0026] Can't start Docker container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (2297777074cc56e7936a6ddb22e42a244f5b86e59d646e343aca746565a67eec):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (ee4f5844ced031650871422c4a76e6668adbdb9b3517a0ffee54f599d14b78ea):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com], try #2 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (f101fd7fb9066c833e50db22e0ef98fae2417e15421b522af1036eece7160ea6):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com], try #2 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (f2e46b88ae648464eea6dc9a94eabfde18d19a8d47209bb6aaed3833917e1090):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com], try #3 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (d3c508aefcaf9bbbae55cb40d5b3dcc18ac74d6560ef76fdc5ad03c51b0795f6):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com], try #3 
WARN[0028] Can't start Docker container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (b77dfa24c0a132287072d17ed1d768174efb6d2c4f9b9dee8163bc650c954e38):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
WARN[0028] Can't start Docker container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (5f588aea1a54b5c1f6ecb640b1518afe2cf8827d54b2e6ef99462e0eabd3ceca):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
FATA[0028] [Failed to start [rke-etcd-port-listener] container on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (2297777074cc56e7936a6ddb22e42a244f5b86e59d646e343aca746565a67eec):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1))]

What is the underlying operating system you are using in this case?

What version of Docker are you using, and how did you install it?

Generally, when this type of error is received, it is due to someone (or something) flushing the iptables rules after Docker has been started. Generally a quick workaround to see if this is the case is to systemctl restart docker to allow Docker to re-add it’s chain.

2 Likes

docker restart fixed it thanks!