Network Bandwidth with Rancher Server

Rancher 1.x
1

We tested in Rancher environment with RancherOS hosts.
We found that network between 2 container in different server has very small Bandwidth
Any ideas for this situation?

There is some test below with iperf.

cross-host (container to container)
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 0.02 GBytes 0.20 Gbits/sec
[ 3] 1.0- 2.0 sec 0.02 GBytes 0.20 Gbits/sec
[ 3] 2.0- 3.0 sec 0.02 GBytes 0.21 Gbits/sec
[ 3] 3.0- 4.0 sec 0.03 GBytes 0.22 Gbits/sec
[ 3] 4.0- 5.0 sec 0.03 GBytes 0.22 Gbits/sec
[ 3] 5.0- 6.0 sec 0.02 GBytes 0.21 Gbits/sec
[ 3] 6.0- 7.0 sec 0.03 GBytes 0.22 Gbits/sec
[ 3] 0.0- 7.0 sec 0.17 GBytes 0.21 Gbits/sec

cross-host (VM to VM)
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 0.11 GBytes 0.96 Gbits/sec
[ 3] 1.0- 2.0 sec 0.11 GBytes 0.95 Gbits/sec
[ 3] 2.0- 3.0 sec 0.11 GBytes 0.94 Gbits/sec
[ 3] 3.0- 4.0 sec 0.11 GBytes 0.94 Gbits/sec
[ 3] 4.0- 5.0 sec 0.11 GBytes 0.95 Gbits/sec
[ 3] 5.0- 6.0 sec 0.11 GBytes 0.94 Gbits/sec
[ 3] 6.0- 7.0 sec 0.11 GBytes 0.94 Gbits/sec
[ 3] 0.0- 7.0 sec 0.77 GBytes 0.94 Gbits/sec

2

Possibly because your VM use the private network with the Gigabits port and Rancher use the public network using a 200Mbits port ?

3

There will of course be some resource overhead introduced by Docker itself (I’d say 10% is a good guide) compared to not using it. Additionally, it’s likely the Charon/Strongswan IPsec encapsulation that occurs for inter-host traffic is having an impact. That encapsulation may also result in NIC offload features not being taken advantage of.

Additionally, the IPsec encapsulation may be causing fragmentation if iperf is using the standard TCP/IP MSS of 1460. I wonder if you can drop it to 1400 and see if things are so bad?

I’m not sure the combination is enough to explain your difference but neither am I sure the iperf tool can provide valid statistics in a container environment.