淺談 Docker 網絡：單節點多容器

時間 2021-01-07

標籤 docker bash 網絡 tcp spa 3d code blog 接口欄目 Docker 简体版

原文原文鏈接

1.同網段多容器訪問

這一節將對 Docker 多容器網絡進行討論，構建容器網絡示意圖以下：

建立容器 demo0 和 demo1:

[root@lianhua ~]$ docker run -it --name demo1 httpd
[root@lianhua ~]$ docker ps
CONTAINER ID        IMAGE      COMMAND             CREATED              STATUS           PORTS            NAMES
39a303a4f993        httpd      "/bin/bash"         About a minute ago   Up 2 seconds                      demo1
7be09e54b24c        httpd      "/bin/bash"         13 hours ago         Up 13 hours                       demo0

查看容器的網絡配置信息：

[root@lianhua ~]$ docker inspect bridge
[
    {
        "Name": "bridge",
        "Id": "1a779d0e62d5a309e1e942862b76d69d4ba9ed9be9c7bcdc051e8de89b0cc3ee",
        "Created": "2020-08-26T00:06:03.910196776+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Containers": {
            "39a303a4f99308e60f7d53a7d471276ce249cc7cf88194d72e2e717cd7533354": {
                "Name": "demo1",
                "EndpointID": "c8c8eeccba33d7fedc09bfca26c0abedcbe53d4ee941fdef29c2e08498f67237",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "7be09e54b24c45100769e131b46259c519710785ccfb68afaa904a1114add9a1": {
                "Name": "demo0",
                "EndpointID": "98399b3c0560aac4ca63de9f79659176562406ac02d917c667852b9a863296bb",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
    }
]

容器間互相訪問：

[root@lianhua ~]$ docker exec -it demo0 /bin/bash
bash-4.2$ ping 172.17.0.3 -c 3
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.047 ms
64 bytes from 172.17.0.3: icmp_seq=3 ttl=64 time=0.044 ms
 
--- 172.17.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.044/0.051/0.063/0.010 ms
 
[root@lianhua ~]$ tcpdump -i docker0 -n icmp -vv    
tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:00:04.074479 IP (tos 0x0, ttl 64, id 44606, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > 172.17.0.3: ICMP echo request, id 22, seq 1, length 64
11:00:04.074518 IP (tos 0x0, ttl 64, id 9717, offset 0, flags [none], proto ICMP (1), length 84)
    172.17.0.3 > 172.17.0.2: ICMP echo reply, id 22, seq 1, length 64
11:00:05.074188 IP (tos 0x0, ttl 64, id 45567, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > 172.17.0.3: ICMP echo request, id 22, seq 2, length 64

容器 demo0 能夠訪問到 demo1，在網橋 docker0 上抓到 ICMP 包。

進一步查看路由表咱們發現這樣一條規則：

[root@lianhua ~]$ iptables -S FORWARD
-A FORWARD -i docker0 -o docker0 -j ACCEPT

該規則定義了連在 docker0 網橋上的容器能夠相互訪問。那麼，若是將 ACCEPT 改爲 DROP 會不會連在 docker0 上的容器相互間沒法訪問呢？咱們改寫規則以下：

[root@lianhua ~]$ iptables -A FORWARD -i docker0 -o docker0 -j DROP
[root@lianhua ~]$ iptables -D FORWARD 19                   # 19 是 -A FORWARD -i docker0 -o docker0 -j ACCEPT 這條規則
[root@lianhua ~]$ iptables -S FORWARD
-A FORWARD -i docker0 -o docker0 -j DROP

規則改寫好了，相互 ping 看可否訪問：

[root@lianhua ~]$ docker exec -it demo0 /bin/bash
bash-4.2$ ping 172.17.0.3 -c 3
--- 172.17.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.044/0.051/0.063/0.010 ms
bash-4.2$ ping 172.17.0.3 -c 3
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
 
--- 172.17.0.3 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
 
[root@lianhua ~]$ tcpdump -i docker0 -n icmp -vv
tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:07:44.129449 IP (tos 0x0, ttl 64, id 44013, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > 172.17.0.3: ICMP echo request, id 23, seq 1, length 64
11:07:45.129195 IP (tos 0x0, ttl 64, id 44650, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > 172.17.0.3: ICMP echo request, id 23, seq 2, length 64
11:07:46.129186 IP (tos 0x0, ttl 64, id 45023, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > 172.17.0.3: ICMP echo request, id 23, seq 3, length 64

和預想的同樣，容器 demo0 沒法訪問 demo1，ICMP 包只有去往 demo1 的，沒有返回的 ICMP 包。

2.不一樣網段多容器訪問

那麼單節點上不一樣網段的多容器互相之間可否訪問呢？基於此設想構建多容器示意圖以下：

bridge1 和 bridge2 是自建的 docker 網絡，它使用的驅動類型是網橋驅動。test1 和 test2 是連到 bridge1 和 bridge2 的容器。

建立 bridge1 和 bridge2 網絡：

[root@lianhua ~]$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
1a779d0e62d5        bridge              bridge              local
a7669a03d994        bridge1             bridge              local
bafee5b9a6d8        bridge2             bridge              local
[root@lianhua ~]$ docker inspect bridge1
[
    {
        "Name": "bridge1",
        "Id": "a7669a03d99463b6bdbca3852a0c9de992e3d1bc66ecfba7e87aa356ca9ed2c5",
        "Created": "2021-01-03T15:59:48.979432719+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
      ...
    }
]

[root@lianhua ~]$ docker inspect bridge2
[
    {
        "Name": "bridge2",
        "Id": "bafee5b9a6d890a2924081146d43fb3a378548e417f595a3a8f5239941178343",
        "Created": "2021-01-03T16:30:47.6901238+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.22.0.0/16",
                    "Gateway": "172.22.0.1"
                }
            ]
        },
      ...
    }
]

show bridge1 和 bridge2 對應的網橋分別爲 br-a7669a03d994 和 br-bafee5b9a6d8。

建立容器 test1 和 test2：

[root@lianhua ~]$ docker run -it --name test1 --network bridge1 httpd
[root@lianhua ~]$ docker run -it --name test2 --network bridge2 httpd
[root@lianhua ~]$ docker inspect bridge1
[
    ...
    {
        "Containers": {
            "861089a681834b16148c36497f689aaaf58fff0937f03bf232cad5bac10bc599": {
                "Name": "test1",
                "EndpointID": "11116435d4a393a8c6673e0ecd1f021729b2d7f652ec1dad088710c7071ce1d9",
                "MacAddress": "02:42:ac:15:00:02",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": ""
            }
        },
    }
]
[root@lianhua ~]$ docker inspect bridge2
[
   ...
   {
        "Containers": {
            "8fd92d473e7b6d30144229bb348f877c7a4dab2fd6285c1c633305d4fe058b35": {
                "Name": "test2",
                "EndpointID": "4bf82b73159c0f86c536d23f7d456dc10bcaee7ec079baf5039966129a0ab816",
                "MacAddress": "02:42:ac:16:00:02",
                "IPv4Address": "172.22.0.2/16",
                "IPv6Address": ""
            }
        },
    }
]

網絡 bridge1 和 bridge2 的子網分別是 172.21.0.0/16 和 172.22.0.0/16，容器 test1 和 test2 分到的 ip 地址是 172.21.0.2 和 172.22.0.2，它們位於不一樣網段。

檢查路由和 ip_forwarding：

[root@lianhua ~]$ ip route
default via 192.168.0.1 dev eth0
169.254.169.254 via 192.168.0.1 dev eth0 proto static
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.21.0.0/16 dev br-a7669a03d994 proto kernel scope link src 172.21.0.1
172.22.0.0/16 dev br-bafee5b9a6d8 proto kernel scope link src 172.22.0.1
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.69
 
[root@lianhua ~]$ cat /proc/sys/net/ipv4/ip_forward
1

路由和 ip_forwarding 是打開的，進到容器 test1 訪問 test2 看看結果:

bash-4.2$ ping 172.22.0.1
PING 172.22.0.1 (172.22.0.1) 56(84) bytes of data.
64 bytes from 172.22.0.1: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 172.22.0.1: icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 172.22.0.1: icmp_seq=3 ttl=64 time=0.036 ms
 
bash-4.2$ ping 172.22.0.2
PING 172.22.0.2 (172.22.0.2) 56(84) bytes of data.
--- 172.22.0.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
 
[root@lianhua ~]$ tcpdump -i br-a7669a03d994 -n icmp -vv
tcpdump: listening on br-a7669a03d994, link-type EN10MB (Ethernet), capture size 262144 bytes
11:42:39.537025 IP (tos 0x0, ttl 64, id 23242, offset 0, flags [DF], proto ICMP (1), length 84)
    172.21.0.2 > 172.22.0.1: ICMP echo request, id 23, seq 1, length 64
11:42:39.537063 IP (tos 0x0, ttl 64, id 7129, offset 0, flags [none], proto ICMP (1), length 84)
    172.22.0.1 > 172.21.0.2: ICMP echo reply, id 23, seq 1, length 64
11:42:40.536198 IP (tos 0x0, ttl 64, id 23603, offset 0, flags [DF], proto ICMP (1), length 84)
    172.21.0.2 > 172.22.0.1: ICMP echo request, id 23, seq 2, length 64
11:42:40.536220 IP (tos 0x0, ttl 64, id 8032, offset 0, flags [none], proto ICMP (1), length 84)
 
[root@lianhua ~]$ tcpdump -i br-a7669a03d994 -n icmp -vv
tcpdump: listening on br-a7669a03d994, link-type EN10MB (Ethernet), capture size 262144 bytes
11:42:54.974063 IP (tos 0x0, ttl 64, id 58344, offset 0, flags [DF], proto ICMP (1), length 84)
    172.21.0.2 > 172.22.0.2: ICMP echo request, id 24, seq 1, length 64
11:42:55.973223 IP (tos 0x0, ttl 64, id 58388, offset 0, flags [DF], proto ICMP (1), length 84)
    172.21.0.2 > 172.22.0.2: ICMP echo request, id 24, seq 2, length 64
11:42:56.973230 IP (tos 0x0, ttl 64, id 58667, offset 0, flags [DF], proto ICMP (1), length 84)
    172.21.0.2 > 172.22.0.2: ICMP echo request, id 24, seq 3, length 64
11:42:57.973190 IP (tos 0x0, ttl 64, id 58989, offset 0, flags [DF], proto ICMP (1), length 84)

奇怪，容器 test1 能夠訪問到 test2 的網關，可是卻訪問不了 test2。查看路由表，咱們找到了這條規則：

-A DOCKER-ISOLATION-STAGE-1 -i br-9c9714b1da04 ! -o br-9c9714b1da04 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-c925ef6b9064 ! -o br-c925ef6b9064 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-2 -o br-9c9714b1da04 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-c925ef6b9064 -j DROP

從網橋 br-9c9714b1da04 發出的包，若是目的接口不是 br-9c9714b1da04，則進入規則 DOCKER-ISOLATION-STAGE-2。而規則 DOCKER-ISOLATION-STAGE-2 的做用是將到網橋 br-9c9714b1da04 和 br-c925ef6b9064 的包丟掉。看到這裏咱們懷疑 test1 到 test2 的包是由於進入了規則 DOCKER-ISOLATION-STAGE-2 而被丟掉了。

改寫 DOCKER-ISOLATION-STAGE-2 規則以下：

[root@lianhua ~]$ iptables -S DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-2 -o br-bafee5b9a6d8 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-2 -o br-a7669a03d994 -j ACCEPT

再一次從 test1 ping test2：

bash-4.2$ ping 172.22.0.2
PING 172.22.0.2 (172.22.0.2) 56(84) bytes of data.
64 bytes from 172.22.0.2: icmp_seq=1 ttl=63 time=0.091 ms
64 bytes from 172.22.0.2: icmp_seq=2 ttl=63 time=0.034 ms
^C
--- 172.22.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.034/0.062/0.091/0.029 ms

修改規則以後 test1 和 test2 能夠互相訪問，驗證了咱們的猜想。

（那麼問題來了，未修改規則以前，爲何訪問 test2 的網關倒是能夠通的呢？經過上面的路由表規則咱們知道訪問網關並無進入規則 DOCKER-ISOLATION-STAGE-2（否則包會被丟掉），因此咱們合理懷疑網關並無建在網橋上。（有知道的童鞋歡迎指出來這個點！））

2.1 多網口容器

除了直接修改路由表實現不一樣網段容器相互訪問的方式以外，也能夠經過在容器中添加另外一張網絡使得容器成爲多網口容器。示意圖以下：

將 test1 接入到網絡 bridge2 上：

[root@lianhua ~]$ docker network connect bridge2 test1
[root@lianhua ~]$ docker inspect bridge2
[
    {
        "Name": "bridge2",
        ...
        "Containers": {
            "459df1132c4b82d8bbde24ecd253d27bb514e3befd9d0c0843aa57257ea08c01": {
                "Name": "test2",
                "EndpointID": "3cd99fae65fbbece78935d5369d3972a13300dd00439ba284793de13b185e530",
                "MacAddress": "02:42:ac:1a:00:02",
                "IPv4Address": "172.26.0.2/16",
                "IPv6Address": ""
            },
            "94a3abcf7e5454b928a562fcf6ebfd33f352d65f1971aee4fb237de0526d16db": {
                "Name": "test1",
                "EndpointID": "8c919123ef44c0b86d2843ab291649ae9283639ec4f02f869efdd1ef73becf18",
                "MacAddress": "02:42:ac:1a:00:03",
                "IPv4Address": "172.26.0.3/16",
                "IPv6Address": ""
            }
        },
    }
]

能夠看到 test1 中有兩個網口，分別配了 bridge1 和 bridge2 的 ip。進入 test1 訪問容器 test2：

bash-4.2$ ping  172.26.0.2
PING 172.26.0.2 (172.26.0.2) 56(84) bytes of data.
64 bytes from 172.26.0.2: icmp_seq=1 ttl=64 time=0.057 ms
64 bytes from 172.26.0.2: icmp_seq=2 ttl=64 time=0.035 ms
64 bytes from 172.26.0.2: icmp_seq=3 ttl=64 time=0.033 ms
64 bytes from 172.26.0.2: icmp_seq=4 ttl=64 time=0.029 ms
64 bytes from 172.26.0.2: icmp_seq=5 ttl=64 time=0.027 ms
64 bytes from 172.26.0.2: icmp_seq=6 ttl=64 time=0.025 ms
 
[root@lianhua ~]$ tcpdump -i br-9c9714b1da04 -n icmp -vv
tcpdump: listening on br-9c9714b1da04, link-type EN10MB (Ethernet), capture size 262144 bytes
23:26:36.616194 IP (tos 0x0, ttl 64, id 4328, offset 0, flags [DF], proto ICMP (1), length 84)
    172.26.0.3 > 172.26.0.2: ICMP echo request, id 34, seq 11, length 64
23:26:36.616221 IP (tos 0x0, ttl 64, id 29472, offset 0, flags [none], proto ICMP (1), length 84)
    172.26.0.2 > 172.26.0.3: ICMP echo reply, id 34, seq 11, length 64
23:26:37.616189 IP (tos 0x0, ttl 64, id 4964, offset 0, flags [DF], proto ICMP (1), length 84)
    172.26.0.3 > 172.26.0.2: ICMP echo request, id 34, seq 12, length 64
23:26:37.616213 IP (tos 0x0, ttl 64, id 29627, offset 0, flags [none], proto ICMP (1), length 84)