Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener

Rancher 2.x
1

not sure what happened where but I am not able to create a cluster with rke, how can I fix this?

(⎈ |k8s.prime.iad0.netskope.com:default)➜  rancher-npe git:(SYS-1307) ✗ ./rke up --config ./rke-cluster.yml
INFO[0000] Running RKE version: v1.0.4                  
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [dialer] Setup tunnel for host [knode3.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [kmaster3.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [kmaster1.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [kmaster2.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [knode1.prime.iad0.netskope.com] 
INFO[0000] [dialer] Setup tunnel for host [knode2.prime.iad0.netskope.com] 
INFO[0001] Checking if container [cluster-state-deployer] is running on host [kmaster3.prime.iad0.netskope.com], try #1 
INFO[0001] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster3.prime.iad0.netskope.com] 
INFO[0002] Starting container [cluster-state-deployer] on host [kmaster3.prime.iad0.netskope.com], try #1 
INFO[0004] [state] Successfully started [cluster-state-deployer] container on host [kmaster3.prime.iad0.netskope.com] 
INFO[0004] Checking if container [cluster-state-deployer] is running on host [knode1.prime.iad0.netskope.com], try #1 
INFO[0005] Image [rancher/rke-tools:v0.1.52] exists on host [knode1.prime.iad0.netskope.com] 
INFO[0006] Starting container [cluster-state-deployer] on host [knode1.prime.iad0.netskope.com], try #1 
INFO[0007] [state] Successfully started [cluster-state-deployer] container on host [knode1.prime.iad0.netskope.com] 
INFO[0009] Checking if container [cluster-state-deployer] is running on host [knode2.prime.iad0.netskope.com], try #1 
INFO[0009] Image [rancher/rke-tools:v0.1.52] exists on host [knode2.prime.iad0.netskope.com] 
INFO[0009] Starting container [cluster-state-deployer] on host [knode2.prime.iad0.netskope.com], try #1 
INFO[0010] [state] Successfully started [cluster-state-deployer] container on host [knode2.prime.iad0.netskope.com] 
INFO[0010] Checking if container [cluster-state-deployer] is running on host [knode3.prime.iad0.netskope.com], try #1 
INFO[0011] Image [rancher/rke-tools:v0.1.52] exists on host [knode3.prime.iad0.netskope.com] 
INFO[0012] Starting container [cluster-state-deployer] on host [knode3.prime.iad0.netskope.com], try #1 
INFO[0013] [state] Successfully started [cluster-state-deployer] container on host [knode3.prime.iad0.netskope.com] 
INFO[0014] Checking if container [cluster-state-deployer] is running on host [kmaster1.prime.iad0.netskope.com], try #1 
INFO[0014] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster1.prime.iad0.netskope.com] 
INFO[0015] Starting container [cluster-state-deployer] on host [kmaster1.prime.iad0.netskope.com], try #1 
INFO[0016] [state] Successfully started [cluster-state-deployer] container on host [kmaster1.prime.iad0.netskope.com] 
INFO[0017] Checking if container [cluster-state-deployer] is running on host [kmaster2.prime.iad0.netskope.com], try #1 
INFO[0018] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster2.prime.iad0.netskope.com] 
INFO[0018] Starting container [cluster-state-deployer] on host [kmaster2.prime.iad0.netskope.com], try #1 
INFO[0019] [state] Successfully started [cluster-state-deployer] container on host [kmaster2.prime.iad0.netskope.com] 
INFO[0019] [certificates] Generating CA kubernetes certificates 
INFO[0019] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0020] [certificates] Generating Kubernetes API server certificates 
INFO[0021] [certificates] Generating Service account token key 
INFO[0021] [certificates] Generating Kube Controller certificates 
INFO[0021] [certificates] Generating Kube Scheduler certificates 
INFO[0021] [certificates] Generating Kube Proxy certificates 
INFO[0022] [certificates] Generating Node certificate   
INFO[0022] [certificates] Generating admin certificates and kubeconfig 
INFO[0022] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0022] [certificates] Generating kube-etcd-kmaster1-prime-iad0-netskope-com certificate and key 
INFO[0023] [certificates] Generating kube-etcd-kmaster2-prime-iad0-netskope-com certificate and key 
INFO[0023] [certificates] Generating kube-etcd-kmaster3-prime-iad0-netskope-com certificate and key 
INFO[0024] Successfully Deployed state file at [./rke-cluster.rkestate] 
INFO[0024] Building Kubernetes cluster                  
INFO[0024] [dialer] Setup tunnel for host [kmaster1.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [kmaster2.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [kmaster3.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [knode2.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [knode3.prime.iad0.netskope.com] 
INFO[0024] [dialer] Setup tunnel for host [knode1.prime.iad0.netskope.com] 
INFO[0025] [network] Deploying port listener containers 
INFO[0025] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster2.prime.iad0.netskope.com] 
INFO[0025] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster3.prime.iad0.netskope.com] 
INFO[0025] Image [rancher/rke-tools:v0.1.52] exists on host [kmaster1.prime.iad0.netskope.com] 
INFO[0025] Starting container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com], try #1 
WARN[0026] Can't start Docker container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (2af2d5c08691e5f1bb1d3a2c69da574ce26dacaad139d0c9ed0e7dc4b7e977ab):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com], try #2 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com], try #1 
WARN[0026] Can't start Docker container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (4afa68ede7789afae27e5b744795c7e65cd5db20cb4e3805fcc2dc4b439138b2):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com], try #3 
INFO[0026] Starting container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com], try #1 
WARN[0026] Can't start Docker container [rke-etcd-port-listener] on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (2297777074cc56e7936a6ddb22e42a244f5b86e59d646e343aca746565a67eec):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (ee4f5844ced031650871422c4a76e6668adbdb9b3517a0ffee54f599d14b78ea):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com], try #2 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (f101fd7fb9066c833e50db22e0ef98fae2417e15421b522af1036eece7160ea6):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com], try #2 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (f2e46b88ae648464eea6dc9a94eabfde18d19a8d47209bb6aaed3833917e1090):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com], try #3 
WARN[0027] Can't start Docker container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (d3c508aefcaf9bbbae55cb40d5b3dcc18ac74d6560ef76fdc5ad03c51b0795f6):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
INFO[0027] Starting container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com], try #3 
WARN[0028] Can't start Docker container [rke-etcd-port-listener] on host [kmaster1.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (b77dfa24c0a132287072d17ed1d768174efb6d2c4f9b9dee8163bc650c954e38):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
WARN[0028] Can't start Docker container [rke-etcd-port-listener] on host [kmaster3.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (5f588aea1a54b5c1f6ecb640b1518afe2cf8827d54b2e6ef99462e0eabd3ceca):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)) 
FATA[0028] [Failed to start [rke-etcd-port-listener] container on host [kmaster2.prime.iad0.netskope.com]: Error response from daemon: driver failed programming external connectivity on endpoint rke-etcd-port-listener (2297777074cc56e7936a6ddb22e42a244f5b86e59d646e343aca746565a67eec):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 1337 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1))]
2

What is the underlying operating system you are using in this case?

What version of Docker are you using, and how did you install it?

Generally, when this type of error is received, it is due to someone (or something) flushing the iptables rules after Docker has been started. Generally a quick workaround to see if this is the case is to systemctl restart docker to allow Docker to re-add it’s chain.

1 Like
3

docker restart fixed it thanks!

4

Hello I seem to have a similar error

FATA[0020] [Failed to start [rke-worker-port-listener] container on host [158.101.199.128]: Error response from daemon: driver failed programming external connectivity on endpoint rke-worker-port-listener (4576203887d3638fd4bd31c44df3594cc17107fa0ce542b241b5225bbc97bb90): (COMMAND_FAILED: ‘/usr/sbin/iptables -w10 -t nat -A DOCKER -p tcp -d 0/0 --dport 10250 -j DNAT --to-destination 172.17.0.2:1337 ! -i docker0’ failed: iptables: No chain/target/match by that name.

however when I look at the iptable rules on my host the chain is indeed there

[root@m2 opc]# sudo iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 30 packets, 1780 bytes)
pkts bytes target prot opt in out source destination
30 1780 DOCKER all – * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 30 packets, 1780 bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2 304 MASQUERADE all – * !docker0 172.17.0.0/16 0.0.0.0/0
152 13916 MASQUERADE all – * enp0s6 0.0.0.0/0 0.0.0.0/0
0 0 MASQUERADE tcp – * * 172.17.0.2 172.17.0.2 tcp dpt:1337
0 0 MASQUERADE tcp – * * 172.17.0.3 172.17.0.3 tcp dpt:1337

Chain OUTPUT (policy ACCEPT 155 packets, 14280 bytes)
pkts bytes target prot opt in out source destination
2 304 DOCKER all – * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all – docker0 * 0.0.0.0/0 0.0.0.0/0
0 0 DNAT tcp – !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2380 to:172.17.0.2:1337
0 0 DNAT tcp – !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2379 to:172.17.0.2:1337
0 0 DNAT tcp – !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6443 to:172.17.0.3:1337

when I try to manually add the rule nothing changes in the table…

Any idea what might be causing this behaviour? Error is occuring during setup of RKE

I am using Oracle linux 8 in oracle cloud ARM instance

docker version is

Docker version 20.10.24, build 297e128

Kubernetes version:

v1.26.4-rancher2-1

rke version:

rke version v1.4.6