[cloud][OVS][sdn] Open vSwitch 初步瞭解

 

What is Open vSwitch?

Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license.  It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).  In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V. See full feature list here

  

Why Open vSwitch ?

https://github.com/openvswitch/ovs/blob/master/Documentation/intro/why-ovs.rst

 

OVN:

http://www.openvswitch.org/support/dist-docs/ovn-architecture.7.html

 CMS |
                                          |
                              +-----------|-----------+
                              |           |           |
                              |     OVN/CMS Plugin    |
                              |           |           |
                              |           |           |
                              |   OVN Northbound DB   |
                              |           |           |
                              |           |           |
                              |       ovn-northd      |
                              |           |           |
                              +-----------|-----------+
                                          |
                                          |
                                +-------------------+
                                | OVN Southbound DB |
                                +-------------------+
                                          |
                                          |
                       +------------------+------------------+
                       |                  |                  | HV 1          |                  |    HV n          |
       +---------------|---------------+  .  +---------------|---------------+
       |               |               |  .  |               |               |
       |        ovn-controller         |  .  |        ovn-controller         |
       |         |          |          |  .  |         |          |          |
       |         |          |          |     |         |          |          |
       |  ovs-vswitchd   ovsdb-server  |     |  ovs-vswitchd   ovsdb-server  |
       |                               |     |                               |
       +-------------------------------+     +-------------------------------+

 

 

 在繼續以前，作好先理解一下 namespace:

[cloud][sdn] network namespace

 

下面兩篇中文介紹，內容類似，寫的都不咋樣。

http://fishcried.com/2016-02-09/openvswitch-ops-guide/ 

https://blog.kghost.info/2014/11/19/openvswitch-internal/

 

這個偏實踐指導，寫的好：

https://www.ibm.com/developerworks/cn/cloud/library/1401_zhaoyi_openswitch/

 

從源碼編譯：

文檔：https://docs.openvswitch.org/en/latest/intro/install/general/

須要注意是否支持內核模塊的編譯，有所不一樣。

[root@D128 thirdparty]# git clone https://github.com/openvswitch/ovs.git
[root@D128 ovs]# git checkout v2.7.0 [root@D128 ovs]# yum install autoconf automake libtool [root@D128 ovs]# ./boot.sh [root@D128 ovs]# ./configure --prefix=/root/BUILD_ovs/ [root@D128 ovs]# make [root@D128 ovs]# make install

上邊是沒編譯內核模塊的。。。

再編個內核模塊吧！！

[root@D128 ovs]# yum install kernel-devel-$(uname -r) [root@D128 ovs]# ./configure --prefix=/root/BUILD_ovs/ --with-linux=/lib/modules/$(uname -r)/build [root@D128 ovs]# uname -a Linux D128 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

kernel版本太舊，編不過。降版本到v2.6.0

[root@D128 ovs]# git checkout v2.6.0 Previous HEAD position was c298ef7... Set release date for 2.7.0. HEAD is now at 7a0f907... Set release date for 2.6.0. [root@D128 ovs]# git branch * (detached from v2.6.0) master [root@D128 ovs]# 

仍是編不過。若是想編譯經過必需要找對彼此兼容的版本。算了直接用centos自帶的ko（反正是學習瞭解階段，希望能兼容跑起來）。。。

[root@D128 ovs]# modprobe openvswitch [root@D128 ovs]# lsmod|grep openvswitch openvswitch 106996  0 nf_nat_ipv6 14131  1 openvswitch nf_nat_ipv4 14115  1 openvswitch nf_defrag_ipv6 35104  2 openvswitch,nf_conntrack_ipv6 nf_nat 26787  3 openvswitch,nf_nat_ipv4,nf_nat_ipv6 nf_conntrack 133387  6 openvswitch,nf_nat,nf_nat_ipv4,nf_nat_ipv6,nf_conntrack_ipv4,nf_conntrack_ipv6 libcrc32c 12644  4 xfs,openvswitch,nf_nat,nf_conntrack [root@D128 ovs]# 

 

運行：

[root@D128 ovs]# export PATH=$PATH:/root/BUILD_ovs/share/openvswitch/scripts/ [root@D128 ~]# ovs-ctl --system-id=random start Starting ovsdb-server [ OK ] Configuring Open vSwitch system IDs [ OK ] Starting ovs-vswitchd [ OK ] Enabling remote OVSDB managers [ OK ] [root@D128 ~]#

數據庫應該已經創建了默認的，而且都初始化了。

 

測試：

[root@D128 BUILD_ovs]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 00:0c:29:2f:cf:32 brd ff:ff:ff:ff:ff:ff [root@D128 BUILD_ovs]# ./bin/ovs-vsctl add-br br0 [root@D128 BUILD_ovs]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 00:0c:29:2f:cf:32 brd ff:ff:ff:ff:ff:ff 3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether 92:e5:c6:d2:ec:a2 brd ff:ff:ff:ff:ff:ff 4: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether be:e8:bd:df:ff:41 brd ff:ff:ff:ff:ff:ff [root@D128 BUILD_ovs]# ./bin/ovs-vsctl add-port br0 ens33 [root@D128 BUILD_ovs]# brctl show bridge name bridge id STP enabled interfaces [root@D128 BUILD_ovs]# 

 

ovs-vsctl add-br 增長的這倆個設備是什麼？

我寫了個腳本去判斷：

[root@D128 ~]# cat ip_link_show_type.sh #! /bin/bash TYPE=" vlan veth vcan dummy ifb macvlan macvtap bridge bond ipoib ip6tnl ipip sit vxlan gre gretap ip6gre ip6gretap vti nlmon bond_slave geneve bridge_slave macsec"

for T in $TYPE do echo $T ip link show type $T done

居然不屬於這裏邊任意一個類型。。。。

 

雖然沒有正面回答，可是看一下下面這個問題的答案，就能很好的理解了：

同時，也講解了KVM的tap設備怎麼可OVS協做，以及須要注意什麼。

https://github.com/openvswitch/ovs/blob/master/Documentation/faq/issues.rst

Q: I created a tap device tap0, configured an IP address on it, and added it to a bridge, like this:

 

大概就是說，OVS的bridge和OVS的internal port是OVS單獨實現的兩種特殊設備。

 

 

報錯：

[root@D128 BUILD_ovs]# ./bin/ovs-vsctl add-port br0 p0 ovs-vsctl: Error detected while setting up 'p0': could not open network device p0 (No such device).  See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/root/BUILD_ovs/var/log/openvswitch". [root@D128 BUILD_ovs]# 

要這樣：

https://github.com/openvswitch/ovs-issues/issues/110

The port‘s name should be a exist interface use ifconfig to see, such as eth0. If you just want to use a virtual port name to make a test you should 
specify the port's type like ovs-vsctl add-port br0 port0 -- set Interface port0 type=internal or ovs-vsctl set Interface port0 type=internal

 

[root@D128 BUILD_ovs]# ./bin/ovs-vsctl add-port br0 port0 -- set Interface port0 type=internal [root@D128 BUILD_ovs]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 00:0c:29:2f:cf:32 brd ff:ff:ff:ff:ff:ff 3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether 92:e5:c6:d2:ec:a2 brd ff:ff:ff:ff:ff:ff 4: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether be:e8:bd:df:ff:41 brd ff:ff:ff:ff:ff:ff 5: port0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether f6:cb:c2:69:fc:e0 brd ff:ff:ff:ff:ff:ff [root@D128 BUILD_ovs]# ./bin/ovs-vsctl show 528b5679-22e8-484b-947b-4499959dc341 Bridge "br0" Port "port0" Interface "port0" type: internal Port "br0" Interface "br0" type: internal ovs_version: "2.7.0" [root@D128 BUILD_ovs]# 

 

查看br0，port0這兩個設備。

[root@D128 BUILD_ovs]# ethtool -i br0 driver: openvswitch version: firmware-version: expansion-rom-version: bus-info: supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no [root@D128 BUILD_ovs]# ethtool -i port0 driver: openvswitch version: firmware-version: expansion-rom-version: bus-info: supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no [root@D128 BUILD_ovs]# 

 

增長namespace

[root@D128 BUILD_ovs]# ip netns add ns0 [root@D128 BUILD_ovs]# ip link set port0 netns ns0

[root@D128 BUILD_ovs]# ip netns exec ns0 ip addr add 192.168.1.100/24 dev port0 [root@D128 BUILD_ovs]# ip netns exec ns0 ifconfig port0 promisc up

查看：

[root@D128 BUILD_ovs]# ./bin/ovs-ofctl show br0 OFPT_FEATURES_REPLY (xid=0x2): dpid:0000bee8bddfff41 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 2(port0): addr:00:00:00:00:0c:e1 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max LOCAL(br0): addr:be:e8:bd:df:ff:41 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 [root@D128 BUILD_ovs]# ./bin/ovs-dpctl show system@ovs-system: lookups: hit:174 missed:44 lost:0 flows: 0 masks: hit:263 total:0 hit/pkt:1.21 port 0: ovs-system (internal) port 1: br0 (internal) port 2: port0 (internal) [root@D128 BUILD_ovs]# 

 

[root@D128 BUILD_ovs]# ip addr add 192.168.1.101/24 dev br0 [root@D128 BUILD_ovs]# ip link set br0 up

 

如今兩個namespace能夠經過br0 互通了。

增長一個of規則

[root@D128 BUILD_ovs]# ./bin/ovs-ofctl add-flow br0 "priority=1 idle_timeout=0, in_port=2,actions=mod_nw_src:9.181.137.1,normal" [root@D128 BUILD_ovs]# ./bin/ovs-ofctl show br0 OFPT_FEATURES_REPLY (xid=0x2): dpid:0000bee8bddfff41 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 2(port0): addr:00:00:00:00:0c:e1 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max LOCAL(br0): addr:be:e8:bd:df:ff:41 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 [root@D128 BUILD_ovs]# ./bin/ovs-

 

抓包能夠看見，原地址已經被修改成9.181.137.1

[root@D128 BUILD_ovs]# tcpdump -i br0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:39:00.988146 IP 9.181.137.1 > localhost: ICMP echo request, id 3101, seq 271, length 64
11:39:01.988227 IP 9.181.137.1 > localhost: ICMP echo request, id 3101, seq 272, length 64
11:39:02.988113 IP 9.181.137.1 > localhost: ICMP echo request, id 3101, seq 273, length 64
11:39:03.988133 IP 9.181.137.1 > localhost: ICMP echo request, id 3101, seq 274, length 64

 

暫時先到這。做爲一個初步瞭解。

 

--------------------------------------------------   update @ 2018-03-30 20：11 -----------------------------------

在OVS port上抓包。

把port流量鏡像出來。

[root@vrouter-ovs ~]# ip link add dev mirror type dummy [root@vrouter-ovs ~]# ip link set mirror up

[root@dr-lb ~]# ovs-vsctl add-port ovs-br0 mirror

[root@vrouter-ovs ~]# ovs-vsctl -- set Bridge br-tun mirror=@mi -- --id=@pmirror get Port mirror -- --id=@patch get Port tun-to-int -- --id=@mi create Mirror name=mymi select-dst-port=@patch select-src-port=@patch output-port=@pmirror [root@vrouter-ovs ~]# tcpdump -i mirror -nn

 

查看和刪除mirror

# ovs-vsctl list Mirror # ovs-vsctl clear bridge ovsbr0 mirrors

 

ovs-vsctl set Bridge ovs-br0 mirrors=@mi -- --id=@pmirror get Port mirror-br0 -- --id=@patch get port vxlanclient0 -- --id=@mi create Mirror name=mymi select-dst-port=@patch select-src-port=@patch output-port=@pmirror