So here is the problem, I have recently updated my server hardware to Dell R710s over the holidays. During this time I got a great deal on 3 PCIe HP NC550SFP 10gb network cards. I have purchased the proper HP compatible SFP DAC (direct access copper) cable to connect the 2 systems together.
After installing SLES 12 and setting it up, I added the 10gb cards in which the OS had no trouble finding. I can configure 1 port on each system with a IP in the 10.10.10.x/24 range.
[SIZE=1]p3p1 Link encap:Ethernet HWaddr B4:99:BA:F8:DB:B8
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Even though from system “A” I can ping system “A” 10gb network IP, when I try pinging system “B” IP, I get the following…
[SIZE=1]
p3p1 Link encap:Ethernet HWaddr B4:99:BA:F8:4B:D8
inet addr:10.10.10.3 Bcast:10.10.10.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:336 (336.0 b)
pinky:~ # ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
From 10.10.10.3 icmp_seq=1 Destination Host Unreachable
From 10.10.10.3 icmp_seq=2 Destination Host Unreachable
From 10.10.10.3 icmp_seq=3 Destination Host Unreachable[/SIZE]
I could use some assistance in troubleshooting this in hopes of getting 10Gb network between my 2 xen VM servers.
I could use some assistance in troubleshooting this in hopes of
getting 10Gb network between my 2 xen VM servers.[/color]
Are both nics connected through a switch?
Can each of them be pinged from a third device?
This is purely a guess since I’m unfamiliar your your equipment…
If they are direct connected, this sounds like what might happneed if
your devices didn’t support Auto-MDIX (automatic medium-dependent
interface crossover) and you didn’t use a crossover cable.
Or, perhaps your cable is defective…
–
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below this post.
Thank you.
This is purely a guess since I’m unfamiliar your your equipment…
If they are direct connected, this sounds like what might happneed if
your devices didn’t support Auto-MDIX (automatic medium-dependent
interface crossover) and you didn’t use a crossover cable.
Defective cable is what crossed my mind first[/color]
Are you using a direct connection between your two servers or are you
using a switch?
–
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below this post.
Thank you.
Defective cable is what crossed my mind first[/color]
So you have:
one cable?
3 2-port nics?
2 servers?
If you had second (good) cable you could substitute it to determine if
the first one is defective.
Failing that…
Are you sure the “active” port on each of the nics are connected?
Maybe one of your nics is defective. Have you tried swapping in your
3rd nic on each server to see if it helps?
–
Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below this post.
Thank you.
could you please also show the output of “ip addr list” and “ip route list” from both machine A and B, so that we can see any possible effects of your routing setup?
I am very sorry to all who have replied. I never received the notification emails and assumed there was no response to my thread. In cleaning up my email junk mail while connected to web mail, I saw a response email.
Again my deepest apologies.
As too the problem, I ended up resolving the issue last weekend. The cards firmware were out of date. I updated the BE2NET driver on my systems which caused the system to become unstable but slowed down the boot screen where it shows all the services starting. Slow enough that I saw text showing an error message about the firmware being too old. As such I fought getting the cards updated to the newest firmware, and now I have communication.
Prior to the firmware, the system would see the cards, and configure the cards. And a system could even ping its own card. But the systems could not ping the other.
Speeds are currently around the same speeds as my 1G Ethernet with a 20GB file transferring between 2 systems at around 75MB/s and completing in 3min 30sec.
I have found that the systems look to be running PCIx 2.0 and so there is probably a 5Gb transfer rate capability between cards, as such I will probably never see 10G transfer rate. But for my needs, a 4G to 5G rate would be good enough.
As too my setup…
2 Dell Gen 11 R710’s (2U)
each server has a HP NC550SFP dual port network card
Purchased HP DAC cable for direct connection between the two servers via SFP port.
Prior to purchasing the hardware, I had confirmed that it was too be Auto-MDIX compliant.
Since getting this running, I have added a 3rd server to act as a ISCSI storage server with the same card. And the 2 other systems only connect to that machine which has both its ports bridged. This will be my VM environment where the file system holding the VM hd files will be ISCSI.
Does anyone here have access to a good Document on DRBD with LVM for use in XEN vm server configurations? I read thru the NFS server with DRBD and LVM document, but it does not feel like a good fit for my needs.
The end goal of these 2 servers and the reason for the 10GB network is combining LVM and DRBD where I can pass drive partitions directly to the VMs as harddrives. And have that replicated between the 2 xen servers allowing me to live migrate the VMs between the 2 xen servers.
In my past setup, the VMs hd’s were RAW files, and I was using SLES 11 sp4 and drbd to replicate between 2 disk. But this time, I want to allow xen vms access directly to the hd. I configured a single drive with LVM and gave that to XEN which did a great job of carving partitions out of the overall disk to give to VMs. But now I need that replicated between the servers with DRBD so the vm can live migrate between the 2. Keep in mind, as things change in the VMs environment, it writes those changes to the HD. And those changes have to be replicated between the 2 xen servers in real time. And each xen host will divide up 8 or 10 vms including some plex media vms.
Meanwhile, my fall back is to run ISCSI which gives me no fail over for my storage and forces my VMs hd’s to be RAW file’s. This I see is going to cause my performance problems. I am not sure if setting up ISCSI and LVM would be a good setup, nor do able.
I believe that this will be too much of a hassle, in the end.
How about setting up the disk images as files on a cluster file system (i.e. OCFS2) above replicated storage (i.e. via DRBD)?
Either way you’ll have to solve some design issues:
What situation do you want to protect from? Disk failure, node failure, both?
How much resources can you spend? Would it be possible to split up storage and compute? Will you need to expand to three or more Xen nodes in the foreseeable future?
How do you want to distribute your compute load? I assume that you’d prefer to run the “Xen part” in active/active mode, that is running VMs on both nodes simultaneously.
How are you going to fail over? Manually or automatically?
I strongly recommend to run Xen with VM locking, to protect from starting the same VM on multiple nodes simultaneously. Been there, done that You’ll need a cluster FS to solve this issue.
Your “change process” will determine on how to bring LVM into the picture, i.e.: If you think about setting up a shared disk via DRB active/active and use LVM on top of that, you cannot simply change the VG on nodeA and have the changes active on nodeB. You’ll need to run something like cLVM for that.
From personal experience, you’ll likely get a more stable solution by running some HAE iSCSI solution (i.e. over a/p DRBD) offering a common disk resource, which you can use on both (and optionally: more/all) Xen servers as an OCFS2 storage device. Putting both services (storage and compute) on the same nodes will, to some extend, complicate things, but as long as that split is very present when you think about your services, you should get along.
Let’s take further discussion to the HAE forum (or possibly to the virtualization forum, if it comes to Xen-specific issues).
You’ll need a cluster FS to solve this issue.