Setting a specific live migration channel

SUSE Linux Enterprise Extensions SLES High Availability Extension
1

Hello All;

SLES 11 SP4 with HA - using KVM hypervisor

The host servers have a 2 x 1Gb nics (bonded and bridged) connected to the LAN and the first cluster comm channel , a second 1Gb nic is a redundant communication channel for Corosync, and 2 x 10Gb iSCSI multipath links to the SAN.

When a live migration takes place, the process takes a long time (minutes), like it is using the LAN for live migration.

Is there a way to specify to use the 10Gb link(s) for live migration (or any migration)?

And how can I check/see which channel/link/NIC is being used for the live migration?

2

Hi johngoutbeck ;),

Is there a way to specify to use the 10Gb link(s) for live migration (or any migration)?

in /etc/xen/xend-config.sxp, there are two options related to that. If they’re not set explicitly, then xend will listen on any interface:

[CODE]# Address xend should listen on for relocation-socket connections, if

xend-relocation-server is set.

Meaning and default as for xend-address above.

Also, interface name is allowed (e.g. eth0) there to get the

relocation address to be bound on.

#(xend-relocation-address ‘’)
[/CODE]
“xend-relocation-address” will let you define on which interface / IP address the local xend will listen for migration requests. So for your environment, set this to the 10G interface to avoid using the 1G network. This is optional

[CODE]# The hosts allowed to talk to the relocation port. If this is empty (the

default), then all connections are allowed (assuming that the connection

arrives on a port and interface on which we are listening; see

xend-relocation-port and xend-relocation-address above). Otherwise, this

should be a space-separated sequence of regular expressions. Any host with

a fully-qualified domain name or an IP address that matches one of these

regular expressions will be accepted.

For example:

(xend-relocation-hosts-allow ‘^localhost$ ^.*\\.example\\.org$’)

#(xend-relocation-hosts-allow ‘’)
#(xend-relocation-hosts-allow ‘^localhost$ ^localhost\\.localdomain$’)[/CODE]
“xend-relocation-hosts-allow” lets you create a list of remote Xen servers that may relocate their VMs to this server… if you’re switching interfaces and have this option activated, make sure the names/patterns will match the 10G interfaces (or rather their IP addresses / DNS names) used by the migration.

Now, when starting the live migration via “xm”, you specify which destination server to use. The man page reads:

[QUOTE] migrate domain-id host [OPTIONS]
Migrate a domain to another host machine. Xend must be running on other host machine, it must be running the same version of Xen, it must have the migration TCP port open and accepting connections from the source host, and there must be sufficient
resources for the domain to run (memory, disk, etc).
[/QUOTE]

It the “host” parameter that defines where the destination Xen server is contacted - you’ll need to specify the IP or DNS name that will resolve to the IP of the 10G interface of the destination server.

If all Xen servers are on the same 10G network, in a common IP subnet, then this should be all. If you have a more complex IP setup, more steps might be required (the issue might be with default routes using the 1G network and having no explicit rules that tell the source Xen server to send the packets via its 10G interface when targeting the 10G IP address of the destination server).

Regards,
J

3

Hi,

use tcpdump to dump traffic on your 10G interface for port 8002 (or port 8003, when using SSL - or on whatever port you configured in /etc/xen/xend-config.sxp) to see if the actual traffic goes via this interface.

Regards,
J

4

Hello JMozdzen;

Thank you for this very good information.

Looks like the setup you describe is for the XEN hypervisor. We’re running the KVM hypervisor on SLES 11 SP4. Will the config files you noted work for KVM or is there a different set?

We are planning to move to the SLES 12 OS with KVM, will the same config files apply to SLES 12 SPx?

Have a good day.

5

Hi,

and “oh my…”, I even remember having read that you’re using KVM… and yes of course, my reply is about Xen :frowning:

How do you issue the migration request? Assuming it is via “virsh”, it’S man page states:

[QUOTE] When migrateuri is not specified, libvirt will automatically determine the hypervisor specific URI. Some hypervisors, including QEMU, have an optional “migration_host” configuration parameter (useful when the host has multiple network interfaces). If
this is unspecified, libvirt determines a name by looking up the target host’s configured hostname.

       There are a few scenarios where specifying migrateuri may help:

       ·   The configured hostname is incorrect, or DNS is broken.  If a host has a hostname which will not resolve to match one of its public IP addresses, then libvirt will generate an incorrect URI.  In this case migrateuri should be explicitly specified,
           using an IP address, or a correct hostname.

       ·   The host has multiple network interfaces.  If a host has multiple network interfaces, it might be desirable for the migration data stream to be sent over a specific interface for either security or performance reasons.  In this case migrateuri
           should be explicitly specified, using an IP address associated with the network to be used.

       ·   The firewall restricts what ports are available.  When libvirt generates a migration URI, it will pick a port number using hypervisor specific rules.  Some hypervisors only require a single port to be open in the firewalls, while others require a
           whole range of port numbers.  In the latter case migrateuri might be specified to choose a specific port number outside the default range in order to comply with local firewall policies.

       See <http://libvirt.org/migration.html#uris> for more details on migration URIs.

[/QUOTE]

So while my original comments were for “xm”, they semantically apply to your situation: Whereever you define the target host for the migration request, make sure to specify it’s 10G interface, likely via an interface-specific DNS name or via IP.

I’ve not used KVM on SLES11SP4 yet, but on later systems, there’s /etc/libvirt/qemu.conf, where you’ll again find similar parameters to the ones I originally described for Xen:

[CODE]# Override the listen address for all incoming migrations. Defaults to

0.0.0.0, or :: if both host and qemu are capable of IPv6.

#migration_address = “0.0.0.0”

The default hostname or IP address which will be used by a migration

source for transferring migration data to this host. The migration

source has to be able to resolve this hostname and connect to it so

setting “localhost” will not work. By default, the host’s configured

hostname is used.

#migration_host = “host.example.com”

Override the port range used for incoming migrations.

Minimum must be greater than 0, however when QEMU is not running as root,

setting the minimum to be lower than 1024 will not work.

Maximum must not be greater than 65535.

#migration_port_min = 49152
#migration_port_max = 49215[/CODE]

It’s “migration_address” that is typically set to “listen on all hosts”, which you might use to limit migrations to 10G interfaces only, and you’ll likely have to give the 10G interface a unique DNS name so that originating servers can reach the KVM target server’s 10G interface - I’d put that DNS name in “migration_host”.

I hope this time my coffee level was high enough, sorry for the misleading initial response.

Regards,
J