Unable to migrate a domain

SUSE Linux Enterprise Server SLES Virtualization
1

Hello,

I’m running SLES 11 SP1 as a XEN host, I have shared NFS storage between two XEN servers. Both XEN servers have access to the NFS share, I’ve created a single test SLES11 guest and am in the final stages of testing XEN as our new virtualization platform. The one step I cannot seem to make work correctly is live migrating a domain to another machine.

I have both XEN servers xend-config.sxp files set to enable migration per much googling. I can connect to both machines using virt-manager (i.e. I can open virt-manager on server A and can add a connection to server B and manage both from that console).

When I attempt to live migrate I receive the following error:

POST operation failed: xend_post: error from xen daemon:
(xend.err “can’t connect: [Errno 104] Connection reset by peer”)

Googling the above results with few other examples of the same error with no resolutions posted. Any ideas on what steps I missed to get live migration working?

Thanks,

2

Hi amginenigma,

that error message sounds to me as if the target xend isn’t properly configured to listen for migration requests.

How are xend-relocation-server, xend-relocation-port, xend-relocation-address, and xend-relocation-hosts-allow set in the config file?
Can you confirm, via i.e. netstat, that xend is actually listening on that relocation port? Is it listening on the interface you’re using to connect both servers?

Regards,
Jens

3

Hi jmozdzen,

From the /etc/xen/xend-config.sxp file:

(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address ‘’)
(xend-relocation-hosts-allow ‘’)

netstat -a | grep 8002 does not show any activity, so the question is what do I need to do to coax XEN into listening.

Thanks,

4

Hi amginenigma,

better use “netstat -an”, there’sa definition for port 8002 in /etc/services.

You might also be able to check via “lsof -i -P|grep 8002” some find that output to be more readable.

Regards,
Jens

5

dangit I always forget the “n”

tcp 0 0 0.0.0.0:8002 0.0.0.0:* LISTEN

So it appears to be listening, where to from here?

6

good to see the port’s open.

can you connect manually, i.e. via “telnet”? If yes, anything to see in the xend.log files on the two hosts? I guess the “sending” xend will mostly complain about the connection reset, hopefully the “receiving” xend is a bit more verbose…

If you cannot connect manually, is there any firewalling active on the “receiving” host?

Regards,
Jens

7

Ok… so telnet shows port is open. There are no firewalls active on this network anywhere.

tail -f on the receiving XEN host gives this when I try to migrate:

[2012-08-23 10:54:31 5581] WARNING (connection:293) Rejected connection from 10.139.1.120 (xen-vm3.doc.nv.gov).

and thanks for the ideas.

8

I suggest that you add that host explicitly to the (currently empty) list of permitted hosts. I know the comment section suggests otherwise, but it may very well be that an empty list means “allow… no-one”.

Regards,
Jens

9

Figured out a work-around. I was host names (which do resolve by the XEN server) I attempted to migrate both with the cli and GUI using IP address and bamo it worked! Not sure why if the server can resolve the hosts the migration command cannot, but meh whatever a win is a win.

10

[QUOTE=jmozdzen;6461]Hi amginenigma,

better use “netstat -an”, there’sa definition for port 8002 in /etc/services.

You might also be able to check via “lsof -i -P|grep 8002” some find that output to be more readable.

Regards,
Jens[/QUOTE]
For posterity: If the -p flag is included in the argument and the command is executed as the root user, netstat will include the name of the process that opened the socket. e.g.,

sles-xen-2:~ # netstat -tanp | grep 631 tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 4604/cupsd tcp 0 0 ::1:631 :::* LISTEN 4604/cupsd

11

[QUOTE=jmozdzen;6459]Hi amginenigma,

that error message sounds to me as if the target xend isn’t properly configured to listen for migration requests.

How are xend-relocation-server, xend-relocation-port, xend-relocation-address, and xend-relocation-hosts-allow set in the config file?
Can you confirm, via i.e. netstat, that xend is actually listening on that relocation port? Is it listening on the interface you’re using to connect both servers?

Regards,
Jens[/QUOTE]
For posterity: From Section 9.3, Migrating Virtual Machines of the the SLES 11 Virtualization with Xen Administration Guide…

[QUOTE]The following xend options, which are located in the /etc/xen/xend-config.sxp file, need to be set on both hosts to make live migration work.

(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address ")
(xend-relocation-hosts-allow ")

For information on modifying xend settings, see Section 5.2, Controlling the Host by Modifying Xend Settings. For more details about using xm to migrate VM Guest systems, see Section 5.6, Migrating Xen VM Guest Systems.[/QUOTE]