容器編排系統k8s之Ingress資源

時間 2020-12-22

標籤 html node linux nginx git github web json 後端 api 欄目 HTML 简体版

原文原文鏈接

　　前文咱們瞭解了k8s上的service資源的相關話題，回顧請參考：http://www.javashuo.com/article/p-xcxfprec-nz.html；今天咱們來了解下k8s上的Ingress資源的相關話題；html

　　咱們知道在k8s上service是用來解決Pod訪問問題，它是經過kube-proxy在每一個節點上建立iptables規則或ipvs規則，在用戶請求某個pod時，用戶的請求會被其service規則所捕獲，從而實現訪問對應pod；對於service來說，用戶請求直接在傳輸層就被捕獲轉發，效率很高效，但這同時也引入了一個新問題；好比咱們運行的pod對外客戶端訪問須要https通訊，若是使用service這種4層調度，那就意味着每一個pod上咱們要配置證書，這很顯然不是咱們想要作的；那有沒有什麼辦法作到在用戶訪問pod對應的service時使用https，而對應pod裏又不用https協議呢？答案是有的；好比咱們可使用nginx來作https會話卸載器；咱們只須要在代理上配置證書便可；又好比咱們在k8s上運行了各類各樣的pod，這些pod的功能每一個都不同，有點是專門處理用戶認證的，有點是專門處理站點主頁的，有點專門處理支付的等等，而這些pod對外都是提供一個獨有的url，那麼這些pod須要怎麼才能被集羣外部訪問到呢？咱們知道對於一個站點來說，若是後端有多個server同時提供一種服務，咱們能夠把這些同功能的server定義成一個組，而後使用nginx代理將不一樣功能url的訪問代理到不一樣組上便可；這樣一來解決了後端多server被負載訪問的問題；那麼對於k8s上這種同功能的pod怎麼歸併成一個組呢？用戶訪問不一樣url怎麼調度到不一樣的組上呢？很顯然要想實現這些功能，在k8s上應該有一個相似nginx同樣的代理存在；這個代理就叫作ingress 控制器；ingress 控制器和k8s上的其餘控制不同，ingress控制器並不能直接運行爲kube-controller-manager的一部分，它相似k8s集羣上的coredns，須要在集羣上單獨部署，本質上就是一個pod，咱們可使用k8s上的ds或deploy控制器來建立它；ingress controller pod的做用主要是引入集羣外部流量，並實時監控着apiserver上ingress資源的變更，並將其ingress中定義的規則轉化爲對應ingress控制器對應應用程序的專有配置，而後動態的重載或重啓對應守護進程來使其配置文件生效；在k8s上ingress是一種標準資源，它本質上就是咱們定義的基於dns名稱（host）或url路徑把請求轉發至指定service資源的規則；簡單講ingress就是咱們用來定義代理的配置所建立的資源；ingress控制器就是把對應ingress規則轉換爲對應ingress控制器中應用程序的專有配置，而後重啓或重載對應配置文件使其生效的組件；node

　　ingress和ingress controller pod的關係linux

　　提示：如上圖所示，ingress就是ingress 控制器pod的代理規則；用戶請求某個後端pod所提供的服務時，首先會經過ingress controller pod把流量引入到集羣內部，而後ingress controller pod根據ingress定義的規則，把對應ingress規則轉化爲對應ingress controller pod實現的對應應用的配置（ingress controller 能夠由任何具備七層反向代理功能的服務實現，好比nginx,haproxy等等）而後再適配用戶請求，把對應請求反代到對應service上；而對於pod的選擇上，ingress控制器能夠基於對應service中的標籤選擇器，直接同pod直接通訊，無須經過service對象api的再次轉發，從而省去了用戶請求到kube-proxy實現的代理開銷（本質上ingress controller 也是運行爲一個pod，和其餘pod在同一網段中）；nginx

　　ingress controller部署git

　　在k8s上ingress controller的實現有不少，好比基於nginx的，基於haproxy的等等，這裏以nginx爲例；github

　　下載ingress-nginx包web

 wget https://github.com/kubernetes/ingress-nginx/archive/nginx-0.28.0.tar.gz

　　解壓包，找到對應的部署清單json

[root@master01 ~]# ll
total 92144
-rw------- 1 root root 65586688 Dec  8 15:16 flannel-v0.13.1-rc1.tar
drwxr-xr-x 2 root root     4096 Dec 21 21:04 manifests
-rw-r--r-- 1 root root 28760559 Dec 21 21:02 nginx-0.28.0.tar.gz
[root@master01 ~]# tar xf nginx-0.28.0.tar.gz 
[root@master01 ~]# ls
flannel-v0.13.1-rc1.tar  ingress-nginx-nginx-0.28.0  manifests  nginx-0.28.0.tar.gz
[root@master01 ~]# cd ingress-nginx-nginx-0.28.0/
[root@master01 ingress-nginx-nginx-0.28.0]# ls
build         code-of-conduct.md  docs    hack      labels.yaml  mkdocs.yml      README.md              SECURITY_CONTACTS  version
Changelog.md  CONTRIBUTING.md     go.mod  images    LICENSE      OWNERS          requirements-docs.txt  test
cmd           deploy              go.sum  internal  Makefile     OWNERS_ALIASES  rootfs                 vendor
[root@master01 ingress-nginx-nginx-0.28.0]# cd deploy/
[root@master01 deploy]# ls
aws        cloud-generic  grafana   prometheus  static                       with-validating-webhook.yaml.tpl
baremetal  cluster-wide   minikube  README.md   validating-webhook.yaml.tpl
[root@master01 deploy]# cd static/
[root@master01 static]# ls
configmap.yaml  mandatory.yaml  namespace.yaml  provider  rbac.yaml  with-rbac.yaml
[root@master01 static]# pwd
/root/ingress-nginx-nginx-0.28.0/deploy/static
[root@master01 static]#

　　提示：資源配置清單在ingress-nginx-nginx-0.28.0/deploy/static下，名爲mandatory.yaml；後端

　　資源配置清單內容api

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container

View Code

　　提示：以上清單主要定義了一個名稱ingress-nginx的名稱空間，在其名稱空間下建立了幾個configmap，最重要的是用deployment建立了一個ingress-nginx pod；

　　這裏說一下，對於ingress-nginx控制器，它本質仍是運行爲一個pod，對於pod來講要想接入外部訪問流量到集羣內部來，有三種方式，一種是使用NodePort類型的service；第二種是使用ds或deploy控制器，在定義pod模板時使用hostPort把pod端口映射到宿主機方式；第三種是定義pod模板時使用hostNetwork，直接共享宿主機網絡名稱空間；以下所示

　　使用專有NodePort service來引入外部流量

　　提示：這種使用deploy控制管理ingress controller pod，若是在pod模板中沒有暴露端口，則須要建立一個service資源來暴露ingress controller pod的端口來引入外部流量到集羣內部；

　　使用ds控制器管理ingress controller pod在pod模板中使用hostPort方式暴露端口

　　提示：使用ds控制器可以保證每一個節點上只運行一個ingress controller,因此咱們能夠把對應ingress controller pod端端口經過端口映射的方式映射到宿主機上的某一固定端口；

　　使用ds控制器在pod模板中使用hostNetwork方式共享宿主機網絡名稱空間

　　提示：共享宿主機網絡名稱空間，也必須使用ds控制器來確保對應每一個節點上只能運行一個ingress controller pod，這樣才能確保每一個ingress controller pod可以正常把端口暴露出去，以供集羣外部客戶端訪問；

　　選擇上述其中一種方式暴露ingress controller pod的端口便可；若是選擇使用ds控制器來暴露端口，咱們就須要修改其對應資源配置清單中的pod模板，以下所示

　　使用ds控制器來管理ingress controller pod在pod模板中使用hostPort方式暴露端口

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              hostPort: 30080
              protocol: TCP
            - name: https
              containerPort: 443
              hostPort: 30443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container

View Code

　　提示：只需把對應控制器類型更改成DaemonSet，在pod模板中spec字段下把replicas去掉；在spec.template.spec.containers.ports字段中加上nodePort字段指定要把容器的端口映射到宿主機上某個端口；若是暴露的端口是非標準端口，在對應k8s集羣外部咱們還須要部署反代，好比使用nginx，haproxy,lvs；

　　使用ds控制器管理ingress controller pod在ds控制器資源配置中使用hostNetwork

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container

View Code

　　提示：把對應控制器類型更改外DaemonSet，在pod模板中spec字段下的replicas字段去掉；在spec.template.spec字段下加上hostNetwork: true便可；以上兩種使用ds控制器管理ingress controller pod也可使用node選擇器，來篩選在某個節點上建立ingress controller pod；

　　使用deploy控制器管理ingress controller pod，就直接應用mandatory.yaml便可

[root@master01 ~]# kubectl apply -f mandatory.yaml 
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
role.rbac.authorization.k8s.io/nginx-ingress-role created
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
limitrange/ingress-nginx created
[root@master01 ~]#

　　查看應用資源清單建立的資源對象

[root@master01 ~]# kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-controller-5466cb8999-4lsjc   1/1     Running   0          80s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-ingress-controller   1/1     1            1           80s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-ingress-controller-5466cb8999   1         1         1       80s
[root@master01 ~]#

　　提示：能夠看到在ingress-nginx名稱空間下建立了一個deploy控制器，對應控制器建立了一個nginx-ingress-controller控制器pod；可是此pod如今不能被外部客戶端訪問到，咱們須要建立一個service來引入外部流量到此pod上；

　　查看pod標籤

[root@master01 ~]# kubectl get pod -n ingress-nginx --show-labels
NAME                                        READY   STATUS    RESTARTS   AGE     LABELS
nginx-ingress-controller-5466cb8999-4lsjc   1/1     Running   0          4m38s   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx,pod-template-hash=5466cb8999
[root@master01 ~]#

　　根據上述標籤來寫一個建立ingress-service資源的配置清單

[root@master01 ~]# cat ingress-nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-svc
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - port: 80
      name: http
      nodePort: 30080
    - port: 443
      name: https
      nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
[root@master01 ~]#

　　提示：以上配置清單主要把知足對應標籤選擇器的pod關聯起來；並把對應pod的80和443端口分別映射到對應主機上的30080和30443端口；

　　應用配置清單

[root@master01 ~]# kubectl apply -f ingress-nginx-service.yaml
service/ingress-nginx-svc created
[root@master01 ~]# kubectl get svc -n ingress-nginx
NAME                TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-svc   NodePort   10.98.4.208   <none>        80:30080/TCP,443:30443/TCP   13s
[root@master01 ~]#

　　訪問集羣任意節點ip的30080和30443端口，看看是否訪問到對應pod？

　　提示：30080是可以正常訪問的，只是它顯示404，是由於咱們沒有對應的主頁；

　　訪問30443端口

　　提示：30443是一個https端口，因此訪問必須用https協議訪問，這裏提示訪問頁面有風險是由於瀏覽器不信任證書引發的，咱們能夠點擊高級，信任證書便可；一樣30443端口也是返回404，是由於沒有主頁的緣由；兩個端口可以正常訪問，說明咱們在k8s上部署的ingress-nginx controller就部署好了；

　　ingress資源的使用

　　在k8s上建立一個deploy控制器，讓其管理2個 ikubernetes/myapp:v1鏡像運行的pod，而後再建立一個對應的service

[root@master01 manifests]# cat myapp-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      rel: stable
  template:
    metadata:
      namespace: default
      labels:
        app: myapp
        rel: stable
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v1

---
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  selector:
    app: myapp
    rel: stable
  ports:
  - name: http
    port: 80
    targetPort: 80
[root@master01 manifests]#

　　提示：一個清單中定義多個資源，須要用「---」來分割資源；

　　應用資源清單

[root@master01 manifests]# kubectl apply -f myapp-demo.yaml
deployment.apps/myapp created
service/myapp created
[root@master01 manifests]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE             NOMINATED NODE   READINESS GATES
myapp-6479b786f5-9d4mh   1/1     Running   0          11s   10.244.2.98   node02.k8s.org   <none>           <none>
myapp-6479b786f5-k252c   1/1     Running   0          11s   10.244.4.20   node04.k8s.org   <none>           <none>
[root@master01 manifests]# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   4h52m
myapp        ClusterIP   10.105.208.218   <none>        80/TCP    21s
[root@master01 manifests]# kubectl describe svc myapp
Name:              myapp
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=myapp,rel=stable
Type:              ClusterIP
IP Families:       <none>
IP:                10.105.208.218
IPs:               10.105.208.218
Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.98:80,10.244.4.20:80
Session Affinity:  None
Events:            <none>
[root@master01 manifests]#

　　建立ingress資源來反代以上資源

　　示例：建立ingress資源

[root@master01 manifests]# cat ingress-myapp.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: /
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

　　提示：建立ingress資源apiVersion的值要寫成extensions/v1beta1，kind爲Ingress；對應metadata中的annotations的配置表示把ingress資源通知給那個類別的ingress controller，若是k8s集羣上有多個類別的ingress controller時，這一項特別有用；在spec字段主要內嵌了三個字段，rules字段用來定義反代規則列表，其值爲一個對象列表；其中rules字段裏主要host和http字段；host用來指定虛擬主機的fqdn名稱，若是不寫表示匹配任意虛擬主機名稱；http是用來定義指向後端的http選擇器列表；其值爲一個對象，裏面只有一個paths字段，用於指定把請求映射到後端的某個路徑；其值爲一個對象列表；對應paths字段中能夠定義path，用來指定映射後端的路徑；backend用於指定後端pod的service，其值爲一個對象；serviceName用於指定對應pod的service名稱；servicePort用於指定後端服務的端口；以上配置表示把www.myapp.com這個虛擬主機的訪問所有反代至服務名稱爲myapp端口爲80的pod上；

　　應用配置清單

[root@master01 manifests]# kubectl apply -f ingress-myapp.yaml 
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp created
[root@master01 manifests]# kubectl get ingress
NAME            CLASS    HOSTS           ADDRESS   PORTS   AGE
ingress-myapp   <none>   www.myapp.com             80      29s
[root@master01 manifests]#

　　查看ingress資源的詳細信息

[root@master01 manifests]# kubectl describe ingress ingress-myapp
Name:             ingress-myapp
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.myapp.com  
                 /   myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:     kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  81s   nginx-ingress-controller  Ingress default/ingress-myapp
[root@master01 manifests]#

　　提示：能夠看到對應知足service名稱爲myapp而且其端口爲80的pod有兩個；

　　進入ingress controller pod裏，看看對應配置文件是否有www.myapp.com的配置？

[root@master01 manifests]# kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-5466cb8999-4lsjc   1/1     Running   0          78m
[root@master01 manifests]# kubectl exec -it -n ingress-nginx pod/nginx-ingress-controller-5466cb8999-4lsjc -- /bin/sh
/etc/nginx $ cd /etc/nginx/
/etc/nginx $ ls
fastcgi.conf            koi-utf                 modsecurity             owasp-modsecurity-crs   uwsgi_params.default
fastcgi.conf.default    koi-win                 modules                 scgi_params             win-utf
fastcgi_params          lua                     nginx.conf              scgi_params.default
fastcgi_params.default  mime.types              nginx.conf.default      template
geoip                   mime.types.default      opentracing.json        uwsgi_params
/etc/nginx $ grep "www.myapp.com" nginx.conf
        ## start server www.myapp.com
                server_name www.myapp.com ;
        ## end server www.myapp.com
/etc/nginx $

　　提示：能夠看到在對應ingress-nginx 控制器pod中可以搜索到www.myapp.com的配置；說明咱們定義的ingress資源已經被ingress-nginx controller 捕獲；

　　用瀏覽器訪問www.myapp.com看看是否可以訪問到內容？

　　提示：使用www.myapp.com訪問，須要確保對應域名可以正常解析到k8s集羣任意一節點上；能夠看到訪問www.myapp.com:30080可以訪問到對應pod內容；

　　刪除ingress代理規則

[root@master01 manifests]# kubectl delete -f ingress-myapp.yaml 
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions "ingress-myapp" deleted
[root@master01 manifests]# kubectl get ingress 
No resources found in default namespace.
[root@master01 manifests]#

　　示例：配置基於url路徑進行流量分發

[root@master01 manifests]# cat ingress-myapp1.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: /bbs
        backend:
          serviceName: myapp
          servicePort: 80
      - path: /blog
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

　　提示：以上配置表示把www.myapp.com/bbs反代到service名稱爲myapp而且端口爲80的pod上；把www.myapp.com/blog反代到ervice名稱爲myapp而且端口爲80的pod上；我這裏是由於k8s上只有這一種應用，生成環境中按照對應的業務邏輯來反代對應url到對應pod上便可；

　　應用配置清單

[root@master01 manifests]# kubectl apply -f ingress-myapp1.yaml 
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp created
[root@master01 manifests]# kubectl get ingress
NAME            CLASS    HOSTS           ADDRESS   PORTS   AGE
ingress-myapp   <none>   www.myapp.com             80      5s
[root@master01 manifests]# kubectl describe ingress ingress-myapp
Name:             ingress-myapp
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.myapp.com  
                 /bbs    myapp:80 (10.244.2.98:80,10.244.4.20:80)
                 /blog   myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:     ingress.kubernetes.io/rewrite-target: /
                 kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  30s   nginx-ingress-controller  Ingress default/ingress-myapp
[root@master01 manifests]#

　　提示：能夠看到對應ingress上就有兩個url分別指向後端service名稱爲myapp端口爲80的pod上；

　　訪問對應url，看看是否訪問到內容？

　　提示：這裏訪問不到內容的緣由是對應pod內部並無對應url的頁面；

　　進入ingress controller pod內部，查看是否有對應配置？

　　提示：能夠看到對應在ingress中定義的配置，都轉爲對應該ingress controller pod中的配置，說明咱們定義基於url分發流量的ingress沒有問題；

　　示例：定義ingress規則基於主機名稱的虛擬主機

[root@master01 manifests]# cat ingress-myapp2.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: 
        backend:
          serviceName: myapp
          servicePort: 80
  - host: blog.myapp.com
    http:
      paths:
      - path: 
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

　　提示：以上配置表示把www.myapp.com這個虛擬主機名稱的訪問流量分發至service名稱爲myapp端口爲80的pod上；把blog.myapp.com的流量分發至至service名稱爲myapp端口爲80的pod上；生成環境按照對應的service名稱來分發便可；

　　應用配置清單

[root@master01 manifests]# kubectl apply -f ingress-myapp2.yaml 
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp created
[root@master01 manifests]# kubectl get ingress
NAME            CLASS    HOSTS                          ADDRESS   PORTS   AGE
ingress-myapp   <none>   www.myapp.com,blog.myapp.com             80      16s
[root@master01 manifests]# kubectl describe ingress ingress-myapp
Name:             ingress-myapp
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host            Path  Backends
  ----            ----  --------
  www.myapp.com   
                     myapp:80 (10.244.2.98:80,10.244.4.20:80)
  blog.myapp.com  
                     myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:      ingress.kubernetes.io/rewrite-target: /
                  kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  32s   nginx-ingress-controller  Ingress default/ingress-myapp
[root@master01 manifests]#

　　驗證配置信息

　　訪問對應虛擬主機，看看是否可以訪問對應pod？

　　提示：能夠看到兩個虛擬主機名稱均可以正常訪問到，對應也作了調度；

　　示例：建立tls類型的ingress資源

　　建立證書

[root@master01 manifests]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
.........................................+++
........+++
e is 65537 (0x10001)
[root@master01 manifests]# openssl req -x509 -key tls.key -out tls.crt -subj /C=CN/ST=SiChuan/L=GuangYuan/O=Test/CN=www.myapp.com -days 3650  
[root@master01 manifests]#

　　提示：以上兩條命令建立了一個名爲tls.key的私鑰和一個自簽名證書，其名爲tls.crt；

　　建立Secret資源

[root@master01 manifests]# kubectl create secret tls www-myapp-com-ingress-secret --cert=./tls.crt --key=./tls.key 
secret/www-myapp-com-ingress-secret created
[root@master01 manifests]# kubectl get secret
NAME                           TYPE                                  DATA   AGE
default-token-xvd4c            kubernetes.io/service-account-token   3      13d
www-myapp-com-ingress-secret   kubernetes.io/tls                     2      21s
[root@master01 manifests]#

　　提示：在ingress控制器上配置https主機時，不能直接使用私鑰和證書文件，而是須要使用secret資源對象來傳遞相關數據；

　　定義tls類型ingress資源清單

[root@master01 manifests]# cat www-myapp-com-ingress-secret.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - www.myapp.com
    secretName: www-myapp-com-ingress-secret
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: / 
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

　　提示：定義tls類型ingress資源清單，須要在spec字段下用tls字段來指定對應主機名稱，以及secret資源對象的名稱；

　　應用資源清單

[root@master01 manifests]# kubectl apply -f www-myapp-com-ingress-secret.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp-tls created
[root@master01 manifests]# kubectl get ingress
NAME                CLASS    HOSTS                          ADDRESS   PORTS     AGE
ingress-myapp       <none>   www.myapp.com,blog.myapp.com             80        31m
ingress-myapp-tls   <none>   www.myapp.com                            80, 443   8s
[root@master01 manifests]# kubectl describe ingress ingress-myapp-tls
Name:             ingress-myapp-tls
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  www-myapp-com-ingress-secret terminates www.myapp.com
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.myapp.com  
                 /   myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:     kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  26s   nginx-ingress-controller  Ingress default/ingress-myapp-tls
[root@master01 manifests]#

　　驗證：訪問對應虛擬主機名稱，看看對應的https端口是否可以正常訪問到內容？

　　提示：能夠看到使用https協議訪問對應的30443端口可以正常訪問到對應後端pod提供的內容；