Rancher Not Using Internal Address for Container Traffic

I’m trying to figure out how to make Rancher use a specific subnet for all traffic that doesn’t need to accessible outside of K8s cluster. The servers my cluster is running on has 3 networks a 10.0.2.0/24 for management of the physical hosts, 10.0.3.0/24 for external docker traffic and a 10.0.4.0/24 for internal docker traffic. The internal network is 10gb and external network is 1gb so I want all the internal traffic using the faster network. This is especially important for stuff like Longhorn that needs to move a lot of data.

My nodes are just docker on bare metal started with a command like this and when I do a tcpdump against the 1gb interface I see all of traffic there.

Note that rancher-private resolves to a 10.0.4.0/24 address and I do see traffic on port 6443 between the rancher agents on the 10gb network but nothing else

sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.3.2 --server https://rancher-private.dev.example.com --token zf4bkvvjkn4q5547gkgc6x8bd5nnl47zthl6t5lmthv7gs4h5q6qzz --ca-checksum 75e28964c7f30bfbb2e3e30e458b557c3d6197664159767356b486a428893c00 --address 10.0.3.11 --internal-address 10.0.4.11 --worker