OpenShift DNS的機制

時間  2019-11-24
標籤 openshift dns 機制 欄目 系統網絡 简体版
原文   原文鏈接

爲何不直接用kube-dns?css

爲何不直接用kube-dns?html

爲何不直接用kube-dns?node

 

感謝各位前輩的專研，在下午有限的時間裏把Openshift DNS的機制理了一下。更詳細的材料你們能夠參考docker

https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/api

https://www.redhat.com/en/blog/red-hat-openshift-container-platform-dns-deep-dive-dns-changes-red-hat-openshift-container-platform-36tomcat

https://www.cnblogs.com/sammyliu/p/10056035.html服務器

本篇主要是基於3.11版本網絡

 

1.DNS架構

 

也就是說全部容器的dns尋址都是經過外部的dnsmasq以及SkyDNS來進行的，不是走容器內部網絡去找kube-dns或者相似的Pod架構

能夠經過命令查看一下dom

  • 在master節點
[root@master dnsmasq.d]# netstat -tunlp|grep 53
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      16864/openshift     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      14937/openshift     
tcp        0      0 10.128.0.1:53           0.0.0.0:*               LISTEN      3191/dnsmasq        
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      3191/dnsmasq        
tcp        0      0 192.168.56.113:53       0.0.0.0:*               LISTEN      3191/dnsmasq

查看具體的進程，注意進程號

# ps -ef|grep openshift

root      14937  14925  0 15:20 ?        00:00:28 openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2
root      16864  16851  6 15:22 ?        00:16:28 openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=2
root      17582  17570  3 15:23 ?        00:09:22 openshift start master controllers --config=/etc/origin/master/master-config.yaml --listen=https://0.0.0.0:8444 --loglevel=2

查看路由信息

[root@master dnsmasq.d]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.56.1    0.0.0.0         UG    100    0        0 enp0s3
10.128.0.0      0.0.0.0         255.252.0.0     U     0      0        0 tun0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.30.0.0      0.0.0.0         255.255.0.0     U     0      0        0 tun0
192.168.56.0    0.0.0.0         255.255.255.0   U     100    0        0 enp0s3

能夠看到10.128.0.0是pod網段，172.17.0.0是SVC網段，192.169.56.0是宿主機網段，每一個網段都啓動了一個dnsmasq,監聽在53端口

 

  • 在node節點
[root@node1 node]# netstat -tunlp|grep 53
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      17134/openshift     
tcp        0      0 10.131.0.1:53           0.0.0.0:*               LISTEN      3243/dnsmasq        
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      3243/dnsmasq        
tcp        0      0 192.168.56.104:53       0.0.0.0:*               LISTEN      3243/dnsmasq       
ps -ef|grep openshift

root      17134  17120  0 15:21 ?        00:00:31 openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2

 

2.DNS的配置信息

Pod中的dns配置會指向Pod所在宿主機IP，配置爲

$ cat /etc/resolv.conf
nameserver 192.168.56.105
search myproject.svc.cluster.local svc.cluster.local cluster.local redhat.com example.com
options ndots:5

192.168.56.105是pod所在的宿主機

 

宿主機的resolv.conf文件

[root@node1 node]# cat /etc/resolv.conf
# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# Generated by NetworkManager
search cluster.local cluster.local example.com
nameserver 192.168.56.104

在部署環境時，會在每一個節點上部署 /etc/NetworkManager/dispatcher.d/99-origin-dns.sh 文件。每當節點上的 NetworkManager 服務啓動時，該文件會被運行。它的任務包括：

 

  • 建立 dnsmasq 配置文件 :
    • node-dnsmasq.conf （沒有）
    • origin-dns.conf  
    • origin-upstream-dns.conf(沒有)
  • 當 NetworkManager 服務啓動時啓動 dnsmasq 服務
  • 設置宿主機的全部默認路由 IP 爲 Dnsmasq 的偵聽IP
  • 修改 /etc/resolv.conf，設置搜索域，以及將宿主機的默認 IP 做爲 nameserver
  • 建立 /etc/origin/node/resolv.conf

origin-dns.conf的配置目錄在/etc/dnsmasq.d/,內容以下

[root@node1 dnsmasq.d]# cat origin-dns.conf 
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=10000
cache-size=10000
bind-dynamic
min-port=1024
except-interface=lo
# End of config

若是有文件origin-upstream-dns.conf ，中定義了上游（upstream） DNS 名字服務器，若是沒有能夠手工建立.

[root@node2 dnsmasq.d]# cat origin-upstream-dns.conf 
server=10.72.17.5
server=10.68.5.26
server=202.96.134.33
server=202.96.128.86

若是須要解析外部域名，是須要在pod運行的宿主機節點上進行建立的。

node-dnsmasq.conf的內容是

server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1

根據前輩SammyTalksAboutCloud的研究，這個已經寫到程序裏面去了。

能夠經過journalctl -u dnsmasq去查看日誌

[root@node2 dnsmasq.d]# journalctl -u dnsmasq
-- Logs begin at Fri 2018-12-28 22:08:53 CST, end at Thu 2019-01-03 20:04:52 CST. --
Dec 28 22:10:48 node2.example.com systemd[1]: Started DNS caching server..
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: started, version 2.76 cachesize 150
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-D
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: DBus support enabled: connected to system bus
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: warning: no upstream servers configured
Dec 28 22:10:48 node2.example.com dnsmasq[3561]: read /etc/hosts - 7 addresses
Dec 28 22:11:59 node2.example.com dnsmasq[3561]: setting upstream servers from DBus
Dec 28 22:11:59 node2.example.com dnsmasq[3561]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Dec 28 22:11:59 node2.example.com dnsmasq[3561]: using nameserver 127.0.0.1#53 for domain cluster.local

根據日誌信息，知道dnsmasq實際把請求轉發給了監聽在127.0.0.1：53上的skyDNS

skyDNS並非做爲一個單獨的進程啓動，而是在啓動網絡

openshift start network --config=/etc/origin/node/node-config.yaml --kubeconfig=/tmp/kubeconfig --loglevel=2

中啓動,SkyDNS 調用 OpenShift API 服務來獲取主機名、IP地址等信息，而後封裝成標準 DNS 記錄並返回給查詢客戶端。 

 

3.DNS在openshift中的配置

  • master
cat /etc/origin/master/master-config.yaml
dnsConfig:
  bindAddress: 0.0.0.0:8053
  bindNetwork: tcp4

bind在每一個ip的8053端口

  • node
cat /etc/origin/node/node-config.yaml
dnsBindAddress: 127.0.0.1:53
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf

 

根據這種機制，service在宿主機範圍內（不只只是在容器中)可以解析

[root@node2 dnsmasq.d]# dig tomcat.myproject.svc.cluster.local

; <<>> DiG 9.9.4-RedHat-9.9.4-72.el7 <<>> tomcat.myproject.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 719
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tomcat.myproject.svc.cluster.local. IN    A

;; ANSWER SECTION:
tomcat.myproject.svc.cluster.local. 30 IN A    172.30.16.194

;; Query time: 0 msec
;; SERVER: 10.0.3.15#53(10.0.3.15)
;; WHEN: Thu Jan 03 20:26:06 CST 2019
;; MSG SIZE  rcvd: 68

可以訪問

[root@node2 dnsmasq.d]# curl  tomcat.myproject.svc:8080



<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <title>Apache Tomcat/8.5.37</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>

 

4.查詢流程圖

 

查看dnsmasq更詳細日誌

vi /etc/dnsmasq.conf 

# For debugging purposes, log each DNS query as it passes through
# dnsmasq.
log-queries
systemctl restart dnsmasq

 

[root@node2 dnsmasq.d]# journalctl -f -u dnsmasq
-- Logs begin at Fri 2018-12-28 22:08:53 CST. --
Jan 03 20:48:12 node2.example.com dnsmasq[33966]: using nameserver 10.72.17.5#53
Jan 03 20:48:12 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 03 20:48:12 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: setting upstream servers from DBus
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 202.96.128.86#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 202.96.134.33#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 10.68.5.26#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 10.72.17.5#53
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 03 20:48:22 node2.example.com dnsmasq[33966]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: query[A] www.baidu.com from 10.0.3.15
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.128.86
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.134.33
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.68.5.26
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.72.17.5
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: query[AAAA] www.baidu.com from 10.0.3.15
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.128.86
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 202.96.134.33
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.68.5.26
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: forwarded www.baidu.com to 10.72.17.5
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.baidu.com is <CNAME>
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.a.shifen.com is 14.215.177.38
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.a.shifen.com is 14.215.177.39
Jan 03 20:48:39 node2.example.com dnsmasq[33966]: reply www.baidu.com is <CNAME>
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程