咱們知道容器之間的通訊,例如上圖中的10.0.9.3與10.0.9.5通訊是經過overlay網絡,是經過一個VXLAN tannel來實現的。web
可是service和service之間通訊是經過VIP實現的。例如client的service與web 的service進行通訊,而web有一個scale,所以client訪問web是經過訪問虛擬IP(VIP)來實現的。那麼VIP是怎麼映射到具體的10.0.9.5或者10.0.9.6呢?這是經過LVS實現的。docker
LVS,Linux Virtual Server。能夠實如今系統級別的負載均衡。shell
咱們能夠在三個節點上任何一個節點訪問80端口均可以訪問到wordpress,這個實現就是IngressNetWork的做用。任何一臺swarm節點上去訪問端口服務時,會經過端口服務經過本節點IPVS(IP Virtual Service),並經過LVS給loadbanlance到真正具備service上面,例如上圖中咱們經過訪問Docker Host3 bash
轉發到另外兩個節點中。網絡
咱們的實驗環境跟上一節同樣,咱們將whoami 的scale 變爲2app
iie4bu@swarm-manager:~$ docker service ls ID NAME MODE REPLICAS IMAGE PORTS h4wlczp85sw5 client replicated 1/1 busybox:1.28.3 9i6wz6cg4koc whoami replicated 3/3 jwilder/whoami:latest *:8000->8000/tcp iie4bu@swarm-manager:~$ docker service scale whoami=2 whoami scaled to 2 overall progress: 2 out of 2 tasks 1/2: running [==================================================>] 2/2: running [==================================================>] verify: Service converged
查看whoami的運行狀況:負載均衡
iie4bu@swarm-manager:~$ docker service ps whoami ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 6hhuf528spdw whoami.1 jwilder/whoami:latest swarm-manager Running Running 17 hours ago 9idgk9jbrlcm whoami.3 jwilder/whoami:latest swarm-worker2 Running Running 16 hours ago
分別運行在swarm-manager和swarm-worker2節點上。curl
在swarm-manager中訪問whoami:tcp
iie4bu@swarm-manager:~$ curl 127.0.0.1:8000 I'm cc9f97cc5056 iie4bu@swarm-manager:~$ curl 127.0.0.1:8000 I'm f47e05019fd9
在swarm-worker1中訪問whoami:wordpress
iie4bu@swarm-worker1:~$ curl 127.0.0.1:8000 I'm f47e05019fd9 iie4bu@swarm-worker1:~$ curl 127.0.0.1:8000 I'm cc9f97cc5056
爲何對於swarm-worker1來說本地並無whoami的service,確能訪問8000端口?
經過iptables能夠看到本地的轉發規則:
iie4bu@swarm-worker1:~$ sudo iptables -nL -t nat [sudo] password for iie4bu: Chain PREROUTING (policy ACCEPT) target prot opt source destination DOCKER-INGRESS all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination DOCKER-INGRESS all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0 MASQUERADE all -- 172.19.0.0/16 0.0.0.0/0 MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:443 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:22 Chain DOCKER (2 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:172.17.0.2:443 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:81 to:172.17.0.2:80 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:23 to:172.17.0.2:22 Chain DOCKER-INGRESS (2 references) target prot opt source destination DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8000 to:172.19.0.2:8000 RETURN all -- 0.0.0.0/0 0.0.0.0/0
咱們看到DOCKER-INGRESS,它的轉發規則是若是咱們訪問tcp8000duank端口,它會轉發到172.19.0.2:8000上,那麼這個172.19.0.2:8000是什麼呢?
咱們先看一下本地的ip:
iie4bu@swarm-worker1:~$ ifconfig br-3f2fc691f5da Link encap:Ethernet HWaddr 02:42:c8:f4:03:ad inet addr:172.18.0.1 Bcast:172.18.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) docker0 Link encap:Ethernet HWaddr 02:42:43:69:b7:60 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 inet6 addr: fe80::42:43ff:fe69:b760/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:83 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:9208 (9.2 KB) docker_gwbridge Link encap:Ethernet HWaddr 02:42:41:bf:4d:15 inet addr:172.19.0.1 Bcast:172.19.255.255 Mask:255.255.0.0 inet6 addr: fe80::42:41ff:febf:4d15/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:42 errors:0 dropped:0 overruns:0 frame:0 TX packets:142 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3556 (3.5 KB) TX bytes:13857 (13.8 KB) ...... ......
咱們能夠看到本地有個網絡是docker_gwbridge,它的ip是172.19.0.1,這個地址與172.19.0.2在同一個網段。因此說咱們能夠猜想172.19.0.2確定是與docker_gwbridge相連的一個網絡。能夠經過brctl show 查看:
iie4bu@swarm-worker1:~$ brctl show bridge name bridge id STP enabled interfaces br-3f2fc691f5da 8000.0242c8f403ad no docker0 8000.02424369b760 no veth75a496d docker_gwbridge 8000.024241bf4d15 no veth400f4b4 veth44af5f8
能夠看到docker_gwbridge有兩個interface,這兩個interface哪一個是呢?
經過docker network ls:
iie4bu@swarm-worker1:~$ docker network ls NETWORK ID NAME DRIVER SCOPE bdf23298113d bridge bridge local 969e60257ba5 docker_gwbridge bridge local cdcffe1b31cb host host local uz1kgf9j6m48 ingress overlay swarm cojus8blvkdo my-demo overlay swarm 3f2fc691f5da network_default bridge local dba4587ee914 none null local
查看docker_gwbridge的詳細信息:
iie4bu@swarm-worker1:~$ docker network inspect docker_gwbridge [ { "Name": "docker_gwbridge", "Id": "969e60257ba50b070374d31ea43a0550d6cd3ae3e68623746642fe8736dee5a4", "Created": "2019-04-08T09:47:59.343371327+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.19.0.0/16", "Gateway": "172.19.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "5559895ccaea972dad4f2fe52c0ec754d2d7c485dceb35083719768f611552e7": { "Name": "gateway_bf5031da0049", "EndpointID": "6ad54f228134798de719549c0f93c804425beb85dd97f024408c4f7fc393fdf9", "MacAddress": "02:42:ac:13:00:03", "IPv4Address": "172.19.0.3/16", "IPv6Address": "" }, "ingress-sbox": { "Name": "gateway_ingress-sbox", "EndpointID": "177757eca7a18630ae91c01b8ac67bada25ce1ea050dad4ac5cc318093062003", "MacAddress": "02:42:ac:13:00:02", "IPv4Address": "172.19.0.2/16", "IPv6Address": "" } }, "Options": { "com.docker.network.bridge.enable_icc": "false", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.name": "docker_gwbridge" }, "Labels": {} } ]
能夠看到與docker_gwbridge相連的container有兩個,分別是gateway_bf5031da0049和gateway_ingress-sbox,而gateway_ingress-sbox的ip正是172.19.0.2。也就是說數據被轉發到這個gateway_ingress-sbox network namespace中去了。
進入gateway_ingress-sbox:
iie4bu@swarm-worker1:~$ sudo ls /var/run/docker/netns 1-cojus8blvk 1-uz1kgf9j6m 44e6e70b2177 b1ba5b4dd9f2 bf5031da0049 ingress_sbox iie4bu@swarm-worker1:~$ sudo nsenter --net=//var/run/docker/netns/ingress_sbox -bash: /home/iie4bu/anaconda3/etc/profiel.d/conda.sh: No such file or directory CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. If your shell is Bash or a Bourne variant, enable conda for the current user with $ echo ". /home/iie4bu/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc or, for all users, enable conda with $ sudo ln -s /home/iie4bu/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh The options above will permanently enable the 'conda' command, but they do NOT put conda's base (root) environment on PATH. To do so, run $ conda activate in your terminal, or to put the base environment on PATH permanently, run $ echo "conda activate" >> ~/.bashrc Previous to conda 4.4, the recommended way to activate conda was to modify PATH in your ~/.bashrc file. You should manually remove the line that looks like export PATH="/home/iie4bu/anaconda3/bin:$PATH" ^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^ root@swarm-worker1:~#
這樣就進入了ingress_sbox的namespace了。查看ip
root@swarm-worker1:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 02:42:0a:ff:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.255.0.3/16 brd 10.255.255.255 scope global eth0 valid_lft forever preferred_lft forever inet 10.255.0.168/32 brd 10.255.0.168 scope global eth0 valid_lft forever preferred_lft forever 15: eth1@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:13:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet 172.19.0.2/16 brd 172.19.255.255 scope global eth1 valid_lft forever preferred_lft forever
發現ip是172.19.0.2。
爲了掩飾LVS,咱們退出ingress_sbox的namespace。在swarm-worker1中安裝ipvsadm,這是lvs的一個管理工具。
iie4bu@swarm-worker1:~$ sudo apt-get install ipvsadm
安裝成功以後,在進入到ingress_sbox中,而後輸入命令iptables -nL -t mangle
root@swarm-worker1:~# iptables -nL -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8000 MARK set 0x100 Chain INPUT (policy ACCEPT) target prot opt source destination MARK all -- 0.0.0.0/0 10.255.0.168 MARK set 0x100 Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination
MARK一行就表示負載均衡。
輸入命令ipvsadm -l
root@swarm-worker1:~# ipvsadm -l IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 256 rr -> 10.255.0.5:0 Masq 1 0 0 -> 10.255.0.7:0 Masq 1 0 0
10.255.0.5 10.255.0.7就是whoami的service地址。
在swarm-manager上查看whoami:
iie4bu@swarm-manager:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cc9f97cc5056 jwilder/whoami:latest "/app/http" 19 hours ago Up 19 hours 8000/tcp whoami.1.6hhuf528spdw9j9pla7l3tv3t iie4bu@swarm-manager:~$ docker exec cc9 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 10.255.0.168/32 brd 10.255.0.168 scope global lo valid_lft forever preferred_lft forever inet 10.0.2.5/32 brd 10.0.2.5 scope global lo valid_lft forever preferred_lft forever 18: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:ff:00:05 brd ff:ff:ff:ff:ff:ff inet 10.255.0.5/16 brd 10.255.255.255 scope global eth0 valid_lft forever preferred_lft forever 20: eth1@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff inet 172.18.0.3/16 brd 172.18.255.255 scope global eth1 valid_lft forever preferred_lft forever 23: eth2@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:00:02:07 brd ff:ff:ff:ff:ff:ff inet 10.0.2.7/24 brd 10.0.2.255 scope global eth2 valid_lft forever preferred_lft forever iie4bu@swarm-manager:~$
whoami的IP正是10.255.0.5。
在swarm-worker2上查看:
iie4bu@swarm-worker2:~$ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f47e05019fd9 jwilder/whoami:latest "/app/http" 20 hours ago Up 20 hours 8000/tcp whoami.3.9idgk9jbrlcm3ufvkmbmvv2t8 633ddfc082b9 busybox:1.28.3 "sh -c 'while true; …" 20 hours ago Up 20 hours client.1.3iv3gworpyr5vdo0h9eortlw0 iie4bu@swarm-worker2:~$ docker exec -it f47 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 10.255.0.168/32 brd 10.255.0.168 scope global lo valid_lft forever preferred_lft forever inet 10.0.2.5/32 brd 10.0.2.5 scope global lo valid_lft forever preferred_lft forever 24: eth2@if25: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:00:02:0d brd ff:ff:ff:ff:ff:ff inet 10.0.2.13/24 brd 10.0.2.255 scope global eth2 valid_lft forever preferred_lft forever 26: eth1@if27: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff inet 172.18.0.4/16 brd 172.18.255.255 scope global eth1 valid_lft forever preferred_lft forever 28: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:ff:00:07 brd ff:ff:ff:ff:ff:ff inet 10.255.0.7/16 brd 10.255.255.255 scope global eth0 valid_lft forever preferred_lft forever
whoami的IP正是10.255.0.7。
所以當咱們的數據包進入到ingress_sbox經過lvs作了一個負載均衡,也就是說咱們訪問8000端口,它會把數據包轉發到10.255.0.5和10.255.0.7作一個負載。而後就會進入到swarm節點中了。