建立kvm
用nova 在openstack 環境中建立3個kvm，baseimage: RHEL 7.1 ppc64le ，都要掛載internal ip 和 floating ipnode
neutron port-create internal_vlan
nova floating-ip-create floating_vlan
nova list
+--------------------------------------+-----------------------------------+--------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------------------------+--------+------------+-------------+--------------------------------------+
| ef1f1a66-1375-4d9a-b99b-580c8340b59d | linzhbj-kvm.ppc64le-keepalived-n0 | ACTIVE | - | Running | ent_vlan=192.168.33.14, 172.16.0.237 |
| 3eb18869-97c5-42cb-94e7-466d52d4199d | linzhbj-kvm.ppc64le-keepalived-n1 | ACTIVE | - | Running | ent_vlan=192.168.33.15, 172.16.0.238 |
| daf4b686-c33d-48e5-89cc-0a0f99d8fb73 | linzhbj-kvm.ppc64le-keepalived-n2 | ACTIVE | - | Running | ent_vlan=192.168.33.91, 172.16.0.239 |
+--------------------------------------+-----------------------------------+--------+------------+-------------+--------------------------------------+
分別建立internal ip 和 floating ip 一會用作總的virtual ipneutron port-list
| 5c09376e-bd65-4aed-9f4e-dd6ba2879a27 | linzhbj_keepalived_vip0 | fa:16:3e:4f:47:21 | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.91"} |
| 0e0c48e6-2e32-48c5-a01d-c059c4931064 | linzhbj_keepalived_vip1 | fa:16:3e:f9:4f:41 | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.14"} |
| b133aa34-a2fc-4c05-a962-18118de6db83 | linzhbj_keepalived_vip2 | fa:16:3e:aa:02:ea | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.15"} |
| 5ffd38c7-9ec3-4661-84f9-048eda70f738 | linzhbj_keepalived_vip_total | fa:16:3e:ea:81:6a | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.92"} |
nova floating-ip-list
+--------------------------------------+--------------+--------------------------------------+---------------+---------+
| Id | IP | Server Id | Fixed IP | Pool |
+--------------------------------------+--------------+--------------------------------------+---------------+---------+
| 0730073b-9758-409f-843a-0f40e8f9b300 | 172.16.0.238 | 3eb18869-97c5-42cb-94e7-466d52d4199d | 192.168.33.15 | ext_net |
| 1c9c3745-d7d4-435c-b8c5-c19d415fde71 | 172.16.0.239 | daf4b686-c33d-48e5-89cc-0a0f99d8fb73 | 192.168.33.91 | ext_net |
| 7d056ad6-b2d9-47ae-92f3-d383d2aab366 | 172.16.1.60 | | 192.168.33.92 | ext_net |
| a948ef38-01b6-4922-8025-067c0c610f17 | 172.16.0.237 | ef1f1a66-1375-4d9a-b99b-580c8340b59d | 192.168.33.14 | ext_net |
+--------------------------------------+--------------+--------------------------------------+---------------+---------+
將floating ip 與internal ip進行掛載，
neutron floatingip-associate 7d056ad6-b2d9-47ae-92f3-d383d2aab366 5ffd38c7-9ec3-4661-84f9-048eda70f738
配置keepalived
進入每一個kvm安裝keepalived，並進行配置啓動
yum install keepalived
編輯keepalived 配置linux
vim /etc/keepalived/keepalived.conf
global_defs
{
notification_email
{
linzhaolover@163.com
}
notification_email_from linzhaolover@163.com
smtp_server 127.0.0.1
stmp_connect_timeout 30
router_id NodeA
}
vrrp_instance lnmp {
state BACKUP
interface eth0
virtual_router_id 100
priority 170
advert_int 5
track_interface {
eth0
}
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.33.92/20 dev eth0
}
}
service keepalived restart
ip addrjson
[root@host-192-168-33-15 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1454 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:aa:02:ea brd ff:ff:ff:ff:ff:ff
inet 192.168.33.15/20 brd 192.168.47.255 scope global dynamic eth0
valid_lft 85999sec preferred_lft 85999sec
inet 192.168.33.92/20 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:feaa:2ea/64 scope link
valid_lft forever preferred_lft forever
查看keepalive 產生的log
tail -f /var/log/messages
三個kvm都進行上面keepalived配置後，測試通不通，
ping 192.168.33.92
竟然不通，看來還缺乏東西，難道少iptable 規則vim
重點 config allow pairs
# neutron port-update <port-uuid> --allowed-address-pairs type=dict list=true mac_address=<mac_address>,ip_address=<ip_cidr> mac_address 是kvm的mac地址， ip_address 是 virtual ip 的地址，這必定不要搞錯了 neutron port-update 5c09376e-bd65-4aed-9f4e-dd6ba2879a27 --name linzhbj_keepalived_vip0 --allowed-address-pairs type=dict list=true mac_address="fa:16:3e:4f:47:21",ip_address=192.168.33.92 neutron port-update 0e0c48e6-2e32-48c5-a01d-c059c4931064 --name linzhbj_keepalived_vip1 --allowed-address-pairs type=dict list=true mac_address="fa:16:3e:f9:4f:41",ip_address=192.168.33.92 neutron port-update b133aa34-a2fc-4c05-a962-18118de6db83 --name linzhbj_keepalived_vip2 --allowed-address-pairs type=dict list=true mac_address="fa:16:3e:aa:02:ea",ip_address=192.168.33.92 在compute node 中check iptables規則 iptables -S | grep 192.168.33.92 -A neutron-openvswi-s0e0c48e6-2 -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:F9:4F:41 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-openvswi-s5c09376e-b -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:4F:47:21 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-openvswi-sb133aa34-a -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:AA:02:EA -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN 這是對應每一個kvm的mac的 iptables rule
總結
openstack中的kvm，要想正常運行，跑相應的服務，必須在 neutron 中配置相應的規則纔可以容許運行。雖然很麻煩，但起到了很好的保護做用；oop
動動腦筋，辦法老是比問題多，O(∩_∩)O~
bug
port-update 的時候會遇到bug
NT-628A69C Policy doesn't allow (rule:update_port and rule:update_port:allowed_address_pairs) to be performed. vim /etc/neutron/policy.json change "update_port:allowed_address_pairs": "rule:admin_or_network_owner", to "update_port:allowed_address_pairs": "rule:admin_or_owner", 不管nova 仍是neutron 都有本身的policy，因此你要設定好才能夠
系統沒有各個kvm iptables 規則
vim /etc/sysctl.conf net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 使配置生效sysctl -p vim /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini [securitygroup] firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver 重啓agent服務 service neutron-plugin-openvswitch-agent restart # iptables -S | grep 192.168.33 -A neutron-openvswi-s0e0c48e6-2 -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:F9:4F:41 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-openvswi-s0e0c48e6-2 -s 192.168.33.14/32 -m mac --mac-source FA:16:3E:F9:4F:41 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-openvswi-s5c09376e-b -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:4F:47:21 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN -A neutron-openvswi-s5c09376e-b -s 192.168.33.91/32 -m mac --mac-source FA:16:3E:4F:47:21 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN