lvs_nat

nat模式環境模擬

1.實驗環境概覽

192.168.1.186   192.168.75.131          雙網卡         dir     調度器(vip)        ipvsadm，雙網卡
192.168.1.185                           單網卡         rs1     真實服務器1      nginx，網關指定調度器(vip)
192.168.1.190                           單網卡         rs2     真實服務器2      nginx，網關指定調度器(vip)
192.168.75.150                          單網卡(雙網卡)    客戶端 (能夠不用配置，直接使用vip測試，能夠跟調度器同樣的配置，方便遠程鏈接)

2.環境準備

• 準備三臺虛擬機(網卡爲橋接模式，單網卡，直接安裝下一步或者克隆；能上網)，客戶端能夠不用配置,關閉iptables和selinux
• 分別修改hostname(vim /etc/sysconfig/network)；分別爲vip，real_server1，real_server2
• 在vip上安裝ipvsadm，兩臺real_server安裝nginx
• 在real_server1上執行 echo "1111111111111master" > /usr/share/nginx/html/index.html，修改網卡網關
• 在real_server2上執行 echo "2222222222222slave" > /usr/share/nginx/html/index.html，修改網卡網關

vip服務器上增長網卡：在vmware上關閉vip，

1. "編輯虛擬機">"添加">"網絡適配器">"下一步">"NAT模式">"完成"
2. "虛擬網絡編輯器">"VMnet8" 點擊下方 NAT設置查看網關(個人是192.168.75.2)
3. 查看"DHCP設置"查看開啓的網段(設置外網網段的設置範圍)
4. 開啓vip，執行vim /etc/udev/rules.d/70-persistent-net.rules，記錄ATTR{address}，即MAC地址
5. cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1
6. vim /etc/sysconfig/network-scripts/ifcfg-eth1
7. 設置ip爲192.168.75.131，網關爲192.168.75.2，HWADDR填寫剛剛記錄的網卡，設置開機啓動，重啓網卡

進行試驗

1.在調度器上新建腳本

vim /usr/local/sbin/lvs_nat.sh

#! /bin/bash
# director 服務器上開啓路由轉發功能: 
echo 1 > /proc/sys/net/ipv4/ip_forward
# 關閉icmp的重定向
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/eth1/send_redirects

iptables -F
iptables -t nat -F
iptables -t nat -X
#iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE
#這條命令並無關聯
ipvsadm -C
#ipvsadm -A -t 192.168.75.131:80 -s rr -p 300   #-p 300   長連接300秒(會致使5分鐘不更換服務器)
ipvsadm -A -t 192.168.75.131:80 -s wrr
#wrr表示權重輪詢，根據權重來輪詢
ipvsadm -a -t 192.168.75.131:80 -r 192.168.1.185:80 -m -w 1
ipvsadm -a -t 192.168.75.131:80 -r 192.168.1.190:80 -m -w 2

2.執行腳本

sh /usr/local/sbin/lvs_nat.sh

3.測試結果

在vip上執行

curl 192.168.75.131

會根據權重來返回結果

抓包結果

[root@real_server1 ~]# tcpdump -nn -i eth0 host 192.168.75.150 and ! port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
01:00:20.379751 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [S], seq 4207417558, win 14600, options [mss 1460,sackOK,TS val 4562638 ecr 0,nop,wscale 6], length 0
01:00:20.379839 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [S.], seq 3392244670, ack 4207417559, win 14480, options [mss 1460,sackOK,TS val 5501088 ecr 4562638,nop,wscale 6], length 0
01:00:20.388310 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 0
01:00:20.388566 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 175
01:00:20.388772 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [.], ack 176, win 243, options [nop,nop,TS val 5501096 ecr 4562642], length 0
01:00:20.389948 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5501098 ecr 4562642], length 245
01:00:20.395679 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4562654 ecr 5501098], length 0
01:00:20.425383 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4562657 ecr 5501098], length 0
01:00:20.426466 IP 192.168.1.190.80 > 192.168.75.150.53919: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5501134 ecr 4562657], length 0
01:00:20.436678 IP 192.168.75.150.53919 > 192.168.1.190.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4562695 ecr 5501134], length 0
01:00:57.176788 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [S], seq 704010524, win 14600, options [mss 1460,sackOK,TS val 4599425 ecr 0,nop,wscale 6], length 0
01:00:57.178908 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [S.], seq 2021861845, ack 704010525, win 14480, options [mss 1460,sackOK,TS val 5533840 ecr 4599425,nop,wscale 6], length 0
01:00:57.187976 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 0
01:00:57.188021 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 175
01:00:57.188359 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [.], ack 176, win 243, options [nop,nop,TS val 5533851 ecr 4599444], length 0
01:00:57.189733 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5533852 ecr 4599444], length 245
01:00:57.200746 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4599454 ecr 5533852], length 0
01:00:57.200787 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4599456 ecr 5533852], length 0
01:00:57.200820 IP 192.168.1.185.80 > 192.168.75.150.53920: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5533861 ecr 4599456], length 0
01:00:57.212718 IP 192.168.75.150.53920 > 192.168.1.185.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4599460 ecr 5533861], length 0
01:02:09.403093 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [S], seq 970535959, win 14600, options [mss 1460,sackOK,TS val 4671656 ecr 0,nop,wscale 6], length 0
01:02:09.403203 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [S.], seq 1701915808, ack 970535960, win 14480, options [mss 1460,sackOK,TS val 5610111 ecr 4671656,nop,wscale 6], length 0
01:02:09.412386 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4671667 ecr 5610111], length 0
01:02:09.412460 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4671669 ecr 5610111], length 175
01:02:09.412507 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [.], ack 176, win 243, options [nop,nop,TS val 5610120 ecr 4671669], length 0
01:02:09.423352 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5610131 ecr 4671669], length 245
01:02:09.434206 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4671689 ecr 5610131], length 0
01:02:09.434294 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4671689 ecr 5610131], length 0
01:02:09.435994 IP 192.168.1.190.80 > 192.168.75.150.53921: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5610143 ecr 4671689], length 0
01:02:09.439232 IP 192.168.75.150.53921 > 192.168.1.190.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4671697 ecr 5610143], length 0

[root@vip ~]# tcpdump -nn -i eth1 host 192.168.75.150 and ! port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
00:58:58.218430 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [S], seq 2818675909, win 14600, options [mss 1460,sackOK,TS val 4481342 ecr 0,nop,wscale 6], length 0
00:58:58.219493 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [S.], seq 4017612900, ack 2818675910, win 14480, options [mss 1460,sackOK,TS val 5419792 ecr 4481342,nop,wscale 6], length 0
00:58:58.223052 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4481347 ecr 5419792], length 0
00:58:58.247015 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4481349 ecr 5419792], length 175
00:58:58.250997 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [.], ack 176, win 243, options [nop,nop,TS val 5419821 ecr 4481349], length 0
00:58:58.251869 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5419824 ecr 4481349], length 245
00:58:58.253316 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4481377 ecr 5419824], length 0
00:58:58.253974 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4481378 ecr 5419824], length 0
00:58:58.263118 IP 192.168.75.131.80 > 192.168.75.150.53918: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5419836 ecr 4481378], length 0
00:58:58.265457 IP 192.168.75.150.53918 > 192.168.75.131.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4481390 ecr 5419836], length 0
00:59:03.219316 ARP, Request who-has 192.168.75.150 tell 192.168.75.131, length 28
00:59:03.221931 ARP, Reply 192.168.75.150 is-at 00:0c:29:5a:c3:3e, length 46
01:00:19.513727 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [S], seq 4207417558, win 14600, options [mss 1460,sackOK,TS val 4562638 ecr 0,nop,wscale 6], length 0
01:00:19.514865 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [S.], seq 3392244670, ack 4207417559, win 14480, options [mss 1460,sackOK,TS val 5501088 ecr 4562638,nop,wscale 6], length 0
01:00:19.520316 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 0
01:00:19.520645 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4562642 ecr 5501088], length 175
01:00:19.525721 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [.], ack 176, win 243, options [nop,nop,TS val 5501096 ecr 4562642], length 0
01:00:19.526217 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5501098 ecr 4562642], length 245
01:00:19.529640 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4562654 ecr 5501098], length 0
01:00:19.559466 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4562657 ecr 5501098], length 0
01:00:19.561567 IP 192.168.75.131.80 > 192.168.75.150.53919: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5501134 ecr 4562657], length 0
01:00:19.570684 IP 192.168.75.150.53919 > 192.168.75.131.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4562695 ecr 5501134], length 0
01:00:24.515461 ARP, Request who-has 192.168.75.150 tell 192.168.75.131, length 28
01:00:24.526927 ARP, Reply 192.168.75.150 is-at 00:0c:29:5a:c3:3e, length 46
01:00:56.305273 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [S], seq 704010524, win 14600, options [mss 1460,sackOK,TS val 4599425 ecr 0,nop,wscale 6], length 0
01:00:56.317205 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [S.], seq 2021861845, ack 704010525, win 14480, options [mss 1460,sackOK,TS val 5533840 ecr 4599425,nop,wscale 6], length 0
01:00:56.321997 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [.], ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 0
01:00:56.322218 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [P.], seq 1:176, ack 1, win 229, options [nop,nop,TS val 4599444 ecr 5533840], length 175
01:00:56.323050 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [.], ack 176, win 243, options [nop,nop,TS val 5533851 ecr 4599444], length 0
01:00:56.324052 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [P.], seq 1:246, ack 176, win 243, options [nop,nop,TS val 5533852 ecr 4599444], length 245
01:00:56.331228 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [.], ack 246, win 245, options [nop,nop,TS val 4599454 ecr 5533852], length 0
01:00:56.331974 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [F.], seq 176, ack 246, win 245, options [nop,nop,TS val 4599456 ecr 5533852], length 0
01:00:56.332944 IP 192.168.75.131.80 > 192.168.75.150.53920: Flags [F.], seq 246, ack 177, win 243, options [nop,nop,TS val 5533861 ecr 4599456], length 0
01:00:56.335788 IP 192.168.75.150.53920 > 192.168.75.131.80: Flags [.], ack 247, win 245, options [nop,nop,TS val 4599460 ecr 5533861], length 0
01:01:01.317186 ARP, Request who-has 192.168.75.150 tell 192.168.75.131, length 28
01:01:01.318083 ARP, Reply 192.168.75.150 is-at 00:0c:29:5a:c3:3e, length 46

nat模式在收到請求後，客戶端包進入調度器後，調度器修改目的地址；

發送給後端real_server處理，real_server處理完後再返回給調度器；

調度器再修改源地址，返回給客戶

爲何在real_server上能接收到另外的服務器包的信息？相互之間會傳播信息（這個不是很理解）

看了抓包後再分析nat與fullnat模式的區別

• 爲何nat模式下須要將real_server的網關修改爲vip的內網地址？
  nat模式下，vip將請求轉發到後端，是經過修改dst來發送給後端real_server。
  
  但real_server返回包卻出現問題，由於real_server是單網卡，須要將包發送給另外一個網段的機器
  
  須要通過網關來調度，但默認網關卻不能實現這個功能，只有被設爲vip的機器能實現。
  
  只有將real_server網關設置成調度器，返回數據包纔會發送給調度器
  
  同時，網關會將返回包的源ip換成vip。(最終目的)
  
  因此要將real_server的網關設置爲vip的內網地址

• fullnat就沒有這個問題
  fullnat模式下，vip的功能是同時改變src跟dst(源地址和目的地址)，那樣就不須要將real_server網關設置成vip的內網地址
  
  由於vip與real_server都在同一網段，因此就不須要修改網關

• fullnat缺點
  RealServer沒法得到用戶IP；淘寶經過叫TOA的方式解決的，
  
  主要原理是：將client address放到了TCP Option裏面帶給後端RealServer，RealServer收到後保存在socket
  
  的結構體裏並經過toa內核模塊hook了getname函數，這樣當用戶調用getname獲取遠端地址時，返回的是保
  
  存在socket的TCPOption的IP. 百度的BVS是經過叫ttm模塊實現的，其實現方式跟toa基本同樣，只是沒有開源。