ifcfg-br-ex for Openstack

SUSE OpenStack Cloud
1

Hi masters,

I would install openstack kilo multi-node in SLE 12, and I have encountered problem when I prepare the settings for ovs.

There are two ifcfg needed for the openvswitch for the openstack br-ex using, ifcfg-br-ex, ifcfg-eth0(which to be used external int for VM instance).

I set the /etc/sysconfig/network/ifcfg-br-ex follow the forum thread that can only one be found about ovs.

https://forums.opensuse.org/showthread.php/497652-OpenVswitch-how-do-I-assign-ip-address-to-ovs-bridge

BOOTPROTO=‘static’
BROADCAST=’’
ETHTOOL_OPTIONS=’’
BRIDGE=‘yes’
BRIDGE_PORTS= ‘eth0 br-ex’
IPADDR=‘192.168.2.51’
MTU=’’
NAME=’’
NETMASK=‘255.255.255.0’
NETWORK=’’
REMOTE_IPADDR=’’
STARTMODE=‘hotplug’
#DHCLIENT_SET_DEFAULT_ROUTE=‘yes’
ONBOOT=‘yes’

and set /etc/sysconfig/network/ifcfg-eth0

BOOTPROTO=‘none’
#BROADCAST=’’
#ETHTOOL_OPTIONS=’’
#IPADDR=‘192.168.2.51’
#MTU=’’
#NAME=’’
#NETMASK=‘255.255.255.0’
#NETWORK=’’
#REMOTE_IPADDR=’’
STARTMODE=‘auto’
#DHCLIENT_SET_DEFAULT_ROUTE=‘yes’
#ONBOOT=‘yes’

But it not work after service network restart, can’t ping other IP in same subnet.

I had tried another setting the official document provide, but not work too.

https://en.opensuse.org/Portal:Wicked/OpenvSwitch

And also search in mailing list of ovs official site, no solution found.

http://openvswitch.org/pipermail/discuss/2014-October/015350.html

I have some experience of openstack kilo deploying in RHEL(SL, Centos)7, but suse’s ifcfg-br-ex seems not like RHEL’s.

Please help me to get the correct ifcfg, or give me some reference.

We are extremely grateful for any reply.

2

lapac1750,

It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.

Has your issue been resolved? If not, you might try one of the following options:

Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.suse.com/faq.php

If this is a reply to a duplicate posting, please ignore and accept our apologies
and rest assured we will issue a stern reprimand to our posting bot.

Good luck!

Your SUSE Forums Team
http://forums.suse.com

3

Does anyone install openstack kilo /openvswitch v2.1.x in SLE 12?

Please kindly share your ifcfg-br-ex and ifcfg-eth0(which port is ovs binded).

Your sharing will help most SLE12 novice learners to setup ovs quickly and easily.

4

I guess you are installing from http://download.opensuse.org/repositories/Cloud:/OpenStack:/Kilo/SLE_12/ - is that right?

What other IP in the same subnet are you trying to ping? Did you try examining network traffic from both sides using wireshark or tcpdump?

You may also find https://github.com/crowbar/barclamp-network/blob/master/chef/cookbooks/network/templates/default/suse-cfg.erb and https://github.com/crowbar/barclamp-network/blob/master/chef/cookbooks/network/recipes/default.rb#L433 informative - this is how that config file is dynamically generated via the SUSE OpenStack Cloud product.

5

Thank you for your rapid reply.

I think maybe it is as same as

zypper addrepo -f obs://Cloud:OpenStack:Kilo/SLE_12 Kilo

The add repo method is provided by openstack official document
http://docs.openstack.org/kilo/install-guide/install/zypper/content/ch_basic_environment.html#basics-packages

[QUOTE=aspiers;28432]
What other IP in the same subnet are you trying to ping? Did you try examining network traffic from both sides using wireshark or tcpdump?[/QUOTE]

I use network node (br-ex have 192.168.2.51, bind the eth0) to ping compute node(eth4 have 192.168.2.52), and use “tcpdump -i eth4 -c2 icmp” in compute node.

0 packets received by filter.

And run “tcpdump -i eth0” in network node when ping is keeping, no pass-through traffic on eth0, except STP packet generated from switch…

Seems like eth0 have not successfully binded with br-ex.

[QUOTE=aspiers;28432]
You may also find https://github.com/crowbar/barclamp-network/blob/master/chef/cookbooks/network/templates/default/suse-cfg.erb and https://github.com/crowbar/barclamp-network/blob/master/chef/cookbooks/network/recipes/default.rb#L433 informative - this is how that config file is dynamically generated via the SUSE OpenStack Cloud product.[/QUOTE]

Good suggestion!!

I must take the time to comprehensively understand.

Thank you for your professional opinion again.

6

Could you provide the output of these commands?

  • ip addr show
  • brctl show
  • ip route
  • iptables -nvL
  • ethtool eth0
7

[QUOTE=aspiers;28446]Could you provide the output of these commands?

  • ip addr show
  • brctl show
  • ip route
  • iptables -nvL
  • ethtool eth0[/QUOTE]

Sorry for not replying sooner because the Chinese Dragon Boat Festival consecutive holidays.

network:~ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fec6:d6a4/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a5 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fec6:d6a5/64 scope link
valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a6 brd ff:ff:ff:ff:ff:ff
inet 172.24.12.148/24 brd 172.24.12.255 scope global eth3
valid_lft forever preferred_lft forever
inet6 fe80::21e:67ff:fec6:d6a6/64 scope link
valid_lft forever preferred_lft forever
6: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 42:3b:6b:9e:c9:ab brd ff:ff:ff:ff:ff:ff
inet 192.168.2.51/24 brd 192.168.2.255 scope global br-ex
valid_lft forever preferred_lft forever
inet6 fe80::403b:6bff:fe9e:c9ab/64 scope link
valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether 1a:f0:aa:f0:ee:51 brd ff:ff:ff:ff:ff:ff

network:~ # brctl show
If ‘brctl’ is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf brctl
network:~ # zypper install brctl
Refreshing service ‘SUSE_Linux_Enterprise_Server_12_x86_64’.
Retrieving repository ‘Kilo’ metadata …[done]
Building repository ‘Kilo’ cache …[done]
Loading repository data…
Reading installed packages…
‘brctl’ not found in package names. Trying capabilities.
No provider of ‘brctl’ found.
Resolving package dependencies…

Nothing to do.

network:~ # ip route
default via 172.24.12.254 dev eth3 proto dhcp
172.24.12.0/24 dev eth3 proto kernel scope link src 172.24.12.148
192.168.2.0/24 dev br-ex proto kernel scope link src 192.168.2.51

network:~ # iptables -nvL
Chain INPUT (policy ACCEPT 994K packets, 230M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 47 – * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 70754 packets, 11M bytes)
pkts bytes target prot opt in out source destination

network:~ # ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

8

[QUOTE=aspiers;28446]Could you provide the output of these commands?

  • brctl show
    [/QUOTE]

I’m not sure what brtcl mean, it is used to show openvswitch state? or with another CLI to show?

network:~ # ovs-vsctl show
219e04fe-ebe3-498c-a8f5-d13db8007c92
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port “eth0”
Interface “eth0”
ovs_version: “2.1.2”
network:~ # ovs-ofctl show br-ex
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000001e67c6d6a3
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(eth0): addr:00:1e:67:c6:d6:a3
config: 0
state: 0
current: 1GB-FD COPPER AUTO_NEG
advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
supported: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
speed: 1000 Mbps now, 1000 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

9

I guess you made a typo, however /sbin/brctl shows the state of your ethernet bridge configuration in the linux kernel, see the manpage for reference.

10

Thank you for your prompt.

I try the cmd in others SLE(I have prepared 3 SLE for openstack 3 nodes topology).

“”"
controller:~ # brctl show
bridge name bridge id STP enabled interfaces

compute:~ # brctl show
bridge name bridge id STP enabled interfaces

network:~ # brctl show
If ‘brctl’ is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf brctl

“”"
I’m exactly sure here is not a typo, because I use the copy & past.

I got confused that my notwork node’s brctl was missing.

The 3 SLE were installed by the same DVD burned from SLE-12-Server-DVD-x86_64-GM-DVD1.iso and have installed same license.

Whatever, my network node just get the brctl through sftp to other node.

“”"
network:~ # brctl show
bridge name bridge id STP enabled interfaces
br-ex 8000.000000000000 no
“”"

11

brctl comes from the bridge-utils package - can you install that?

12

No, network node has not installed bridge-utils, I think this is crux.

I just run “zypper install bridge-utils”.

“”"
network:~ # brctl show
bridge name bridge id STP enabled interfaces
br-ex 8000.000000000000 no
“”"

Thank you for your very prompt response~!

13

Shouldn’t that be state UP not state UNKNOWN?

14

I have revised the setting, and service network restart.

“”"
network:~ # vi /etc/sysconfig/network/ifcfg-br-ex
BOOTPROTO=‘static’
BROADCAST=‘’
DEVICETYPE=‘ovs’
TYPE=‘OVSBridge’
ETHTOOL_OPTIONS=‘’
BRIDGE=‘yes’
BRIDGE_PORTS= ‘eth0’
BRIDGE_STP=‘off’
BRIDGE_FORWARDDELAY=‘0’
IPADDR=‘192.168.2.51’
MTU=‘’
NAME=‘’
NETMASK=‘255.255.255.0’
NETWORK=‘’
REMOTE_IPADDR=‘’
STARTMODE=‘auto’
#DHCLIENT_SET_DEFAULT_ROUTE=‘yes’
#ONBOOT=‘yes’
HOTPLUG=‘no’

“”"

“”"
network:~ # ip addr show
8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 2a:d5:07:d6:f2:48 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.51/24 brd 192.168.2.255 scope global br-ex
valid_lft forever preferred_lft forever
inet6 fe80::28d5:7ff:fed6:f248/64 scope link
valid_lft forever preferred_lft forever
“”"

The state is still UNKNOWN.

I have seen some example of bridge setting are
BRIDGE_PORTS= ‘eth0’

but the other one is
BRIDGE_PORTS= ‘eth0 br-ex’

Which one is correct setting?

15

Seems that you have two overlapping br-ex bridges: one via Linux bridge (i.e. native, shown by brctl) and one via openvSwitch (shown by ovs-vsctl). You shouldn’t have both.

BTW this is exactly the kind of grief which SUSE OpenStack Cloud eliminates by doing it all for you :wink:

16

[QUOTE=aspiers;28521]Seems that you have two overlapping br-ex bridges: one via Linux bridge (i.e. native, shown by brctl) and one via openvSwitch (shown by ovs-vsctl). You shouldn’t have both.

BTW this is exactly the kind of grief which SUSE OpenStack Cloud eliminates by doing it all for you ;-)[/QUOTE]

I try “brctl show” in the other openstack kilo network node which was deployed by RDO on RHEL 7.0.
And compare it with my SLE openstack kilo network node’s.

“”"
[root@netwo ~]# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.525400223125 yes virbr0-nic

'“”

So, I use the"brctl delbr br-ex" to delete the dummy br setting on SLE, and service openvswitch restart, service network restart.

But still doesn’t work, ping gateway fail.

Would please help me to parse these command results again?

network:~ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
link/ether 00:1e:67:c6:d6:a3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fec6:d6a4/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a5 brd ff:ff:ff:ff:ff:ff
inet6 fe80::21e:67ff:fec6:d6a5/64 scope link
valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1e:67:c6:d6:a6 brd ff:ff:ff:ff:ff:ff
inet 172.24.12.148/24 brd 172.24.12.255 scope global eth3
valid_lft forever preferred_lft forever
inet6 fe80::21e:67ff:fec6:d6a6/64 scope link
valid_lft forever preferred_lft forever
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether 1a:f0:aa:f0:ee:51 brd ff:ff:ff:ff:ff:ff
11: br-ex: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether 00:1e:67:c6:d6:a3 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.51/24 brd 192.168.2.255 scope global br-ex
valid_lft forever preferred_lft forever
inet6 fe80::4873:25ff:fe71:ff96/64 scope link
valid_lft forever preferred_lft forever

network:~ # brctl show
bridge name bridge id STP enabled interfaces

network:~ # ip route
default via 172.24.12.254 dev eth3 proto dhcp
172.24.12.0/24 dev eth3 proto kernel scope link src 172.24.12.148
192.168.2.0/24 dev br-ex proto kernel scope link src 192.168.2.51

network:~ # iptables -nvL
Chain INPUT (policy ACCEPT 1631K packets, 322M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 47 – * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 85938 packets, 14M bytes)
pkts bytes target prot opt in out source destination

network:~ # ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

network:~ # ovs-vsctl show
219e04fe-ebe3-498c-a8f5-d13db8007c92
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port “eth0”
Interface “eth0”
ovs_version: “2.1.2”

network:~ # ovs-vsctl list-ports br-ex
eth0
network:~ # ovs-vsctl list-ifaces br-ex
eth0

network:~ # ovs-vsctl iface-to-br eth0
br-ex

network:~ # ovs-ofctl show br-ex
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000001e67c6d6a3
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
1(eth0): addr:00:1e:67:c6:d6:a3
config: 0
state: 0
current: 1GB-FD COPPER AUTO_NEG
advertised: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
supported: 10MB-HD 10MB-FD 100MB-HD 100MB-FD 1GB-FD COPPER AUTO_NEG AUTO_PAUSE
speed: 1000 Mbps now, 1000 Mbps max
LOCAL(br-ex): addr:00:1e:67:c6:d6:a3
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
network:~ # ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=69250.566s, table=0, n_packets=738, n_bytes=61689, idle_age=0, hard_age=65534, priority=0 actions=NORMAL

network:~ # ovs-ofctl dump-ports br-ex
OFPST_PORT reply (xid=0x2): 2 ports
port 1: rx pkts=687278, bytes=72309453, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=19, bytes=942, drop=0, errs=0, coll=0
port LOCAL: rx pkts=20, bytes=1032, drop=0, errs=0, frame=0, over=0, crc=0
tx pkts=566, bytes=38869, drop=0, errs=0, coll=0

Thanks your eye for detail.

BTW there are no many people whose have SUSE OpenStack Cloud successfully installed would share his ifcfg…Q_Q

17

Hi lapac1750,

[message deleted - I didn’t notice that this was already covered in a separate branch of this thread.]

Regards,
Jens