High network latency between domUs on SLES 11 SP1 xen host

SUSE Linux Enterprise Server SLES Virtualization
1

Hi,
I see very high network latency between domUs running on the same
dom0.
dom0 is SLES 11 SP1 + xen 4.0.2, domUs are SLES 10 SP1 or SLES 10 SP4.
Problem does not occur between dom0 and domUs, or between domUs running
on different servers.
There’s also no problem when dom0 is running with SLES 10 SP4.

ping between 2 domUs:

ping -c10 -s1000 10.1.74.30

PING 10.1.74.30 (10.1.74.30) 1000(1028) bytes of data.
1008 bytes from 10.1.74.30: icmp_seq=1 ttl=64 time=7.82 ms
1008 bytes from 10.1.74.30: icmp_seq=2 ttl=64 time=1.92 ms
1008 bytes from 10.1.74.30: icmp_seq=3 ttl=64 time=5.34 ms
1008 bytes from 10.1.74.30: icmp_seq=4 ttl=64 time=5.32 ms
1008 bytes from 10.1.74.30: icmp_seq=5 ttl=64 time=5.33 ms
1008 bytes from 10.1.74.30: icmp_seq=6 ttl=64 time=1.38 ms
1008 bytes from 10.1.74.30: icmp_seq=7 ttl=64 time=0.143 ms
1008 bytes from 10.1.74.30: icmp_seq=8 ttl=64 time=1.39 ms
1008 bytes from 10.1.74.30: icmp_seq=9 ttl=64 time=1.40 ms
1008 bytes from 10.1.74.30: icmp_seq=10 ttl=64 time=1.40 ms

— 10.1.74.30 ping statistics —
10 packets transmitted, 10 received, 0% packet loss, time 9034ms
rtt min/avg/max/mdev = 0.143/3.150/7.824/2.429 ms

ping between domU and dom0:

ping -c10 -s1000 10.1.74.84

PING 10.1.74.84 (10.1.74.84) 1000(1028) bytes of data.
1008 bytes from 10.1.74.84: icmp_seq=1 ttl=64 time=0.075 ms
1008 bytes from 10.1.74.84: icmp_seq=2 ttl=64 time=0.081 ms
1008 bytes from 10.1.74.84: icmp_seq=3 ttl=64 time=0.079 ms
1008 bytes from 10.1.74.84: icmp_seq=4 ttl=64 time=0.078 ms
1008 bytes from 10.1.74.84: icmp_seq=5 ttl=64 time=0.092 ms
1008 bytes from 10.1.74.84: icmp_seq=6 ttl=64 time=0.090 ms
1008 bytes from 10.1.74.84: icmp_seq=7 ttl=64 time=0.088 ms
1008 bytes from 10.1.74.84: icmp_seq=8 ttl=64 time=0.076 ms
1008 bytes from 10.1.74.84: icmp_seq=9 ttl=64 time=0.077 ms
1008 bytes from 10.1.74.84: icmp_seq=10 ttl=64 time=0.072 ms

The latency also seems to depend on the package size:

ping -c10 -s96 10.1.74.30

PING 10.1.74.30 (10.1.74.30) 96(124) bytes of data.
104 bytes from 10.1.74.30: icmp_seq=1 ttl=64 time=0.101 ms
104 bytes from 10.1.74.30: icmp_seq=2 ttl=64 time=0.133 ms
104 bytes from 10.1.74.30: icmp_seq=3 ttl=64 time=0.138 ms
104 bytes from 10.1.74.30: icmp_seq=4 ttl=64 time=0.132 ms
104 bytes from 10.1.74.30: icmp_seq=5 ttl=64 time=0.131 ms
104 bytes from 10.1.74.30: icmp_seq=6 ttl=64 time=0.140 ms
104 bytes from 10.1.74.30: icmp_seq=7 ttl=64 time=0.145 ms
104 bytes from 10.1.74.30: icmp_seq=8 ttl=64 time=0.146 ms
104 bytes from 10.1.74.30: icmp_seq=9 ttl=64 time=0.147 ms
104 bytes from 10.1.74.30: icmp_seq=10 ttl=64 time=0.145 ms

— 10.1.74.30 ping statistics —
10 packets transmitted, 10 received, 0% packet loss, time 8998ms
rtt min/avg/max/mdev = 0.101/0.135/0.147/0.019 ms

ping -c10 -s97 10.1.74.30

PING 10.1.74.30 (10.1.74.30) 97(125) bytes of data.
105 bytes from 10.1.74.30: icmp_seq=1 ttl=64 time=7.79 ms
105 bytes from 10.1.74.30: icmp_seq=2 ttl=64 time=5.31 ms
105 bytes from 10.1.74.30: icmp_seq=3 ttl=64 time=1.34 ms
105 bytes from 10.1.74.30: icmp_seq=4 ttl=64 time=5.34 ms
105 bytes from 10.1.74.30: icmp_seq=5 ttl=64 time=5.33 ms
105 bytes from 10.1.74.30: icmp_seq=6 ttl=64 time=5.34 ms
105 bytes from 10.1.74.30: icmp_seq=7 ttl=64 time=2.95 ms
105 bytes from 10.1.74.30: icmp_seq=8 ttl=64 time=1.34 ms
105 bytes from 10.1.74.30: icmp_seq=9 ttl=64 time=5.34 ms
105 bytes from 10.1.74.30: icmp_seq=10 ttl=64 time=1.28 ms

— 10.1.74.30 ping statistics —
10 packets transmitted, 10 received, 0% packet loss, time 9034ms
rtt min/avg/max/mdev = 1.288/4.141/7.791/2.136 ms

Is this a bug? Or a (mis)configuration issue? Did anyone else
experienced the same problem?

–
-veit-

-veit-'s Profile: http://forums.novell.com/member.php?userid=12556
View this thread: http://forums.novell.com/showthread.php?t=447140

2

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Not perfect duplication, but my SLES 11 (no SP) system isn’t seeing
this, though perhaps it’s new with a later Xen version. ab20 is a SLES
11 SP1 domU, ab12 is a SLES 10 SP-something domU, and ab1 is the dom0
for both:

novell@ab20:~> ping ab12.lab.novell.com
PING ab12.lab.novell.com (151.155.214.155) 56(84) bytes of data.
64 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=1 ttl=64
time=3.57 ms
64 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=2 ttl=64
time=0.097 ms
64 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=3 ttl=64
time=0.110 ms
64 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=4 ttl=64
time=0.101 ms
64 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=5 ttl=64
time=0.100 ms
64 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=6 ttl=64
time=0.123 ms

Please ignore the total time… having name resolution problems causing
an initial delay getting the IP address.

Good luck.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOoWOkAAoJEF+XTK08PnB5lq0P/i3mK1uR2IRuUXzaQGn51VoI
pkB233UG3FSulfOL/B56HiJggKzzKZTOKDWmLX1cjkSAOqp3ZIhl1aSbx2VK4PQ/
hqukxSz/U+fJXjHxknyKEcQx6uWuoT98diI5IW3fzD7etc4XxtNnARj0Sv7hsjBW
CY3+H32TdbsTXYxM9yNHotUAPzgrwjQ7zpT9WQOT9C07UfSKyR4tHdEWHQHkS6e0
RLoAFxQvchDvrZrisJV5d9o+N+5zai9uiBAF/mwafYsdTcx3PFbyplvqqsG2Y22H
1b4upnWkggLRVmMGR9fbtKwtfj9/jZwYr2WK+d+FIAcfRunTMf5Y3BTYHuIUI/8U
c8RsYWQF5FS94hpWJrsR2+iIV7aiRYjvImj/wEjnQT69rlaAZZDMYeng2P1mezAt
qtnyXqNDVlAoOX9hTkjOzOCifuoCCBuvFwOd/K7lFP76fvRQBH/l0wwqenYu3YOf
QMqYM6Jfng7idiNa6HGkmE/oiMjB+T8NNXjJYcKISSWPZHuSQJNyvvcfIQ0FFyB8
KBnoCcacbDmDprRLbsg/VtW8eK0dlAiMukL5BVfJABdeNdrDIAWOKtMG3j0R61kw
j+aEwHVgaHJ47kI1fWuzr6MmPP0oW8kPPf+x26Giq4RW8Tgau44F4utoyJjfGTfB
8lz/a5iW9Bn2ju3rwqog
=3S9Q
-----END PGP SIGNATURE-----

3

ab;2147772 Wrote:[color=blue]

novell@ab20:~> ping ab12.lab.novell.com
[/color]

Please try the ping again with a larger packet size than the default 64
bytes.
The problem occurs only with packets larger than 104 bytes.

–
-veit-

-veit-'s Profile: http://forums.novell.com/member.php?userid=12556
View this thread: http://forums.novell.com/showthread.php?t=447140

4

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

No luck.

novell@ab20:~> ping -s 1024 ab12.lab.novell.com
PING ab12.lab.novell.com (151.155.214.155) 1024(1052) bytes of data.
1032 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=1 ttl=64
time=1.28 ms
1032 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=2 ttl=64
time=0.133 ms
1032 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=3 ttl=64
time=0.168 ms
1032 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=4 ttl=64
time=0.109 ms
1032 bytes from ab12.lab.novell.com (151.155.214.155): icmp_seq=5 ttl=64
time=0.118 ms

novell@ab20:~> ping -s 1024 ab1.lab.novell.com
PING ab1.lab.novell.com (151.155.214.144) 1024(1052) bytes of data.
1032 bytes from ab1.lab.novell.com (151.155.214.144): icmp_seq=1 ttl=64
time=1.26 ms
1032 bytes from ab1.lab.novell.com (151.155.214.144): icmp_seq=2 ttl=64
time=0.080 ms
1032 bytes from ab1.lab.novell.com (151.155.214.144): icmp_seq=3 ttl=64
time=0.109 ms
1032 bytes from ab1.lab.novell.com (151.155.214.144): icmp_seq=4 ttl=64
time=0.070 ms
1032 bytes from ab1.lab.novell.com (151.155.214.144): icmp_seq=5 ttl=64
time=0.077 ms

May still be the SP/Xen version differences between our systems, maybe.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOpWU8AAoJEF+XTK08PnB5xSsP/it2WDp3IRga/ewo2QvJyu8L
dNE+9BcFWCgLYNNRbfssDSbIO0CxJXNJPQrbgy3PCphYa/edAREGG7N+SCUkZXIa
YImtaAqVjwEe9J2TqHXlTo4fX4j5yuOZxFs2KUoSgVf1k9eQREFFx7TYJtx2BIno
TPKEqWJ9pPX6RLLxoA4C67XY6AXY8a1sYmpSFkn7zDAKeU+s5Q/0s+sMgsAQQ1dP
4JUmdLKgix6akQGUjs/pwx3s+D3McyXqgt9DBsWheP6S/Mrmo5eJQrQgdeKucBc9
xFQ7GpwmMjkRI34TVAD67aJ51C6aA8BomAonFZlMiNVTygfuJzKSurJX3emodwrT
rvitEHcsmuQV2+q7U/PcWepATrxc/mzxHgTEKbFpGD98gOQCLRnVkbTvlKxId4Dj
VNKudSOLKyMB/64E2CO5A7lY42mrk8i5TVD+uY+kzKFruD8J/XceZg3u/iKXwuz8
ZyMJEv9NMWoTzKARYtzO1SXqULar4WcfIFDPMlaTAeB8SHnRuoczf0uQTkLEXkBA
/CkF/UnHz1MxXMtYu7cpYRye8huSH3uHXCHMCsRT9H/ZI1X2A8qTP+6IbKnpXgU3
dv8Q3tqFc53OHz3tth9w3pIlHc2Kc9PJ2myck6Xi1DFR4ceHXZ+i4oneo1NlHWme
fow9mdm56K8+dtjH38xT
=oYui
-----END PGP SIGNATURE-----

5

Am I the only one who is running SLES 11 SP1 + XEN on dom0?
If not, please try to reproduce this on your system. Just do a simple
“ping -s1000” between two domUs.
It would help to know, if this is a general or an individual problem.
Thanks

–
-veit-

-veit-'s Profile: http://forums.novell.com/member.php?userid=12556
View this thread: http://forums.novell.com/showthread.php?t=447140

6

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Found a coworker running Xen on SLES 11 SP1. Using a SLES 11 SP1 domU
and a SLES 10 SP-something domU (pinging FROM the 11 domUs where i have
access to them) I can duplicate what you’re saying pretty reliably.
Bug# 727909 has been submitted with my details including a link to this
thread.

Good luck.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOsbkiAAoJEF+XTK08PnB5JaUP/j4EupaMPyl2XJQc7wb09n6z
lcWuxkAnXFMRcuEk1ojwdeepv6ltNY0Vg6HEk6e4ytEkAfQQ2TIxZiI2swqApSeF
xQ6wkbewftHJ57ImRlpTXOwqtr0N0eBbtmuBhaCTwnDzIlH1WGokwNOAU9NNcG0i
p1RcKDETr1JpB/cLky+zS8CN0a4U1G+lbyc6NccOxU7bvo7DTraOl8IoXSillfwL
8UccL2e24CwvX0RMQtGGeE1Hq7PuPj6BPN8sure6+cC7d5rHQp08yCJuNS5dOcEC
fnsUKNv0yTRvICmerJGVbk7HrN853xXrn5MMR0tv+WmDNigm8jOSNQWBdogsSYmR
XW0vC/uIggLKCRXlGu4ULS1C3Ey5tfShng8U37VQs50JmFuMtKyu9w/J5T4F1bgU
4sLKu1uSwMueXKmm5dOHt/Eop020s0SaUYgalHFWvbtlkndREC2FipjkNq/ax+6s
ZQMT1W07cJrzVdWvNAao+2t+2j4GqDIj62JGcEYyr/d01evssZcd4AsKEmLdg2yF
Nz7CAGp+9EtpFdsmuX5d6RXgYkf6WjB/tWZQWv1Gbd3hUk0KGz7ET+Y/TddMXFBR
9mMEamQtPA0MNWfQJ7xNpnGsryC/mu6EeQ1D/bVvpqW7BP2EUN3PRIgj/eYZ8THv
bPE6st3e+uRkr+PmCdsA
=1IJH
-----END PGP SIGNATURE-----

7

ab;2150907 Wrote:[color=blue]

Found a coworker running Xen on SLES 11 SP1. Using a SLES 11 SP1 domU
and a SLES 10 SP-something domU (pinging FROM the 11 domUs where i
have
access to them) I can duplicate what you’re saying pretty reliably.
Bug# 727909 has been submitted with my details including a link to
this
thread.[/color]
Hi ab, thanks for your efforts!

–
-veit-

-veit-'s Profile: http://forums.novell.com/member.php?userid=12556
View this thread: http://forums.novell.com/showthread.php?t=447140

8

ab;2150907 Wrote:[color=blue]

Bug# 727909 has been submitted with my details including a link to this
thread.[/color]

Hi ab, unfortunately I’m not authorized to access bug #727909. How can
I find out if there’s any progress on this issue?

–
-veit-

-veit-'s Profile: http://forums.novell.com/member.php?userid=12556
View this thread: http://forums.novell.com/showthread.php?t=447140

9

You’re welcome to e-mail me your novell.com-associated e-mail address and
I can add you to the CC list but the SUSE folks often remove those before
they go into super fixing mode, so don’t be offended if that happens. In
the end if you open a Service Request with Novell you can link that to the
bug so that you get updates via Novell/SUSE directly so if that’s an
option it’s probably the “best” way.

Good luck.

10

-veit-;2150845 Wrote:[color=blue]

Am I the only one who is running SLES 11 SP1 + XEN on dom0?
If not, please try to reproduce this on your system. Just do a simple
“ping -s1000” between two domUs.
It would help to know, if this is a general or an individual problem.
Thanks[/color]

We have different sites running SLES 11 SP1 clusters, haven’t seen this
before.

I’ll give this a swing tomorrow.

As far as the domU’s… these are all running paravirtual, which guest
OS and 32 or 64 bit?

Also curious, if you disable tso on the network interface of the domUs
(note: changing the setting will give you a very shot communication
dip)… does that make any difference? (‘ethtool -K ethX tso off/on’ to
set , ‘ethtool -k ethX’ to view settings )

-Willem

–
Novell Knowledge Partner (voluntary sysop)

It ain’t anything like Harry Potter… but you gotta love the magic IT
can bring to this world

magic31’s Profile: http://forums.novell.com/member.php?userid=2303
View this thread: http://forums.novell.com/showthread.php?t=447140

11

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

TID# 7009790 has been published for this issue with a brief explanation
of a workaround (that is not recommended) and a little bit of
information on the reason why this is happening.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOzP5sAAoJEF+XTK08PnB5zTQP/2J5LEBcOmocYo8gwlxc3wxR
HE8km1ZsTIa4+283mN+07ydVcntsxo1oHX376+xzzfCOZ+Tzu1O3MMGteSjyOijs
QbBkofiIbv1AwkE6wP0dMGqbeJtBfATewPxCFiFDEknTRFo+iaDgVCDdkZ6qXU9b
DRL+WSo9NCjUgfO1byemhM56DgN7CKKdVpvSJk98gxjsOIdUaanTMAzvBbhYqmIu
7kQUlnHlI+StupI0+pzGpUtgm1d1ZPiarYQgTW0iMaGwEKppuPWpQ+4dp5QXtSr1
9r1uYMdQutV0foFR7QEdeWaHcb7ayDDIRt6ypQdguSoVUfL6zK49DBwMn7+UBadp
B9HQad4KuMX36FmxQJ/6yxiwXHbkc6OTJf4Xtcj26O/XkwW7RBd6qHbJMWMaVP1p
+HFXWTOiTBLCHYvU3mcsql5tCjbZFvPzMkCujHjcT7aQmiprFVQ25rIiwd6VmMdJ
VwagzD2H7laAXfRGAHJQfFfLPIlJD1Y3eVT8PEGr4ohhNs9X9o24R57a/rNK5S7j
qAzh6Wg0GbTN1YTwJ4/rpqD4jT4ZMNLrQUSPhIazSN8EX5nlWunVNYvS9d9zoDkQ
CP0MPJ53VpZA6tRSvkGnxY22Yt281kB9VFYxa/iGipg3CtsRLHxjpk0oZXJzfsgG
VRMYIyqQf7q4qQGtIUa9
=DJ6Y
-----END PGP SIGNATURE-----

12

ab;2155946 Wrote:[color=blue]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

TID# 7009790 has been published for this issue with a brief
explanation
of a workaround (that is not recommended) and a little bit of
information on the reason why this is happening.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - ‘Enigmail: A simple interface for
OpenPGP email security’ (http://enigmail.mozdev.org/)

iQIcBAEBAgAGBQJOzP5sAAoJEF+XTK08PnB5zTQP/2J5LEBcOmocYo8gwlxc3wxR
HE8km1ZsTIa4+283mN+07ydVcntsxo1oHX376+xzzfCOZ+Tzu1O3MMGteSjyOijs
QbBkofiIbv1AwkE6wP0dMGqbeJtBfATewPxCFiFDEknTRFo+iaDgVCDdkZ6qXU9b
DRL+WSo9NCjUgfO1byemhM56DgN7CKKdVpvSJk98gxjsOIdUaanTMAzvBbhYqmIu
7kQUlnHlI+StupI0+pzGpUtgm1d1ZPiarYQgTW0iMaGwEKppuPWpQ+4dp5QXtSr1
9r1uYMdQutV0foFR7QEdeWaHcb7ayDDIRt6ypQdguSoVUfL6zK49DBwMn7+UBadp
B9HQad4KuMX36FmxQJ/6yxiwXHbkc6OTJf4Xtcj26O/XkwW7RBd6qHbJMWMaVP1p
+HFXWTOiTBLCHYvU3mcsql5tCjbZFvPzMkCujHjcT7aQmiprFVQ25rIiwd6VmMdJ
VwagzD2H7laAXfRGAHJQfFfLPIlJD1Y3eVT8PEGr4ohhNs9X9o24R57a/rNK5S7j
qAzh6Wg0GbTN1YTwJ4/rpqD4jT4ZMNLrQUSPhIazSN8EX5nlWunVNYvS9d9zoDkQ
CP0MPJ53VpZA6tRSvkGnxY22Yt281kB9VFYxa/iGipg3CtsRLHxjpk0oZXJzfsgG
VRMYIyqQf7q4qQGtIUa9
=DJ6Y
-----END PGP SIGNATURE-----[/color]

OH NO! That string of horror is back again Ab! :wink:

Thanks for updating the thread. I had done some tests but have not
seen this. Apparently when running recently patched SLES 10 SP3 and
SLES 10 SP4 (with or without OES on top) domU’s, this latency does not
show itself. At least, I’m seeing no issues here (two different sites
running SLES 11 SP1 as Xen base).

Cheers,
Willem

–
Novell Knowledge Partner (voluntary sysop)

It ain’t anything like Harry Potter… but you gotta love the magic IT
can bring to this world

magic31’s Profile: http://forums.novell.com/member.php?userid=2303
View this thread: http://forums.novell.com/showthread.php?t=447140