I measured inter-host inter-container communication performance with iperf.
In my environment rancher “managed” network performance between containers on different hosts estimates to 25% of raw connection.
Do your CPUs support AES-NI (and is it passed through BIOS and any virtualization)? Since we switched from 3DES to AES (a while ago, v0.1x) there should be minimal overhead on a modern CPU, up to around 2gbps, which seems to be the limit of aesni on typical cloud hosting Intel CPUs.