k8s部署高可用Ingress

時間 2019-11-18

標籤 k8s 部署可用 ingress 简体版

原文原文鏈接

部署高可用Ingress

官網地址https://kubernetes.github.io/ingress-nginx/deploy/html

獲取ingress的編排文件node

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yamlnginx

增長節點標籤git

kubectl label node k8snode1 ingresscontroller=true
kubectl get nodes --show-labelsgithub

附：apache

刪除標籤
kubectl label node k8snode1 ingresscontroller-

更新標籤
kubectl label node k8snode1 ingresscontroller=false --overwrite

一、修改Deployment爲DaemonSet，並註釋掉副本數vim

kind: DaemonSet
  #replicas: 1

二、啓用hostNetwork網絡，並指定運行節點後端

hostNetwork暴露ingress-nginx controller的相關業務端口到主機，這樣node節點主機所在網絡的其餘主機，均可以經過該端口訪問到此應用程序。api

nodeSelector指定以前添加ingresscontroller=true標籤的node網絡

hostNetwork: true
      nodeSelector:
        ingresscontroller: 'true'

三、修改鏡像地址
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.24.1

四、修改容器端口

args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=88 #默認80
            - --https-port=4433 #默認443
            
            
          ports:
            - name: http
              containerPort: 88 #更改成88
            - name: https
              containerPort: 4433 #更改成4433

五、增長master節點容忍

tolerations: #增長容忍，可分配到master節點
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"

六、mandatory.yaml以下
vim mandatory.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      nodeSelector:
        ingresscontroller: 'true'
      tolerations: #增長容忍，可分配到master節點
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=88 #默認80
            - --https-port=4433 #默認443
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 88 #更改成88
            - name: https
              containerPort: 4433 #更改成4433
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

測試

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-static
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-static
  labels:
    name: nginx-static
spec:
  replicas: 1
  template:
    metadata:
      labels:
       name: nginx-static
    spec:
      containers:
       - name: nginx-static
         image: nginx:latest
         volumeMounts:
          - mountPath: /etc/localtime
            name: vol-localtime
            readOnly: true

         ports:
          - containerPort: 80
      volumes:
         - name: vol-localtime
           hostPath:
            path: /etc/localtime
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-static
  labels:
   name: nginx-static
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    name: http
  selector:
    name: nginx-static

--- 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: submodule-checker-ingress
spec:
  rules:
  - host: nginx.weave.pub
    http:
      paths:
      - backend:
          serviceName: nginx-static
          servicePort: 80

kubectl create -f nginx-static.yaml

訪問

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.x.x.x master1 nginx.test.pub
10.x.x.x master2

[root@master1 ingress]# curl nginx.test.pub
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>

1.1 Ingress簡介

1.1.1需求

coredns是實現pods之間經過域名訪問，若是外部須要訪問service服務，需訪問對應的NodeIP:Port。可是因爲NodePort須要指定宿主機端口，一旦服務多起來，多個端口就難以管理。那麼，這種狀況下，使用Ingress暴露服務更加合適。

1.1.2 Ingress組件說明

使用Ingress時通常會有三個組件：反向代理負載均衡器、Ingress Controller、Ingress

一、反向代理負載均衡器

反向代理負載均衡器很簡單，說白了就是 nginx、apache 等中間件，新版k8s已經將Nginx與Ingress Controller合併爲一個組件，因此Nginx無需單獨部署，只須要部署Ingress Controller便可。在集羣中反向代理負載均衡器能夠自由部署，可使用 Replication Controller、Deployment、DaemonSet 等等方式

二、Ingress Controller

Ingress Controller 實質上能夠理解爲是個監視器，Ingress Controller 經過不斷地跟 kubernetes API 打交道，實時的感知後端 service、pod 等變化，好比新增和減小 pod，service 增長與減小等；當獲得這些變化信息後，Ingress Controller 再結合下文的 Ingress 生成配置，而後更新反向代理負載均衡器，並刷新其配置，達到服務發現的做用

三、Ingress

Ingress 簡單理解就是個規則定義；好比說某個域名對應某個 service，即當某個域名的請求進來時轉發給某個 service;這個規則將與 Ingress Controller 結合，而後 Ingress Controller 將其動態寫入到負載均衡器配置中，從而實現總體的服務發現和負載均衡

總體關係以下圖所示：

從上圖中能夠很清晰的看到，實際上請求進來仍是被負載均衡器攔截，好比 nginx，而後 Ingress Controller 經過跟 Ingress 交互得知某個域名對應哪一個 service，再經過跟 kubernetes API 交互得知 service 地址等信息；綜合之後生成配置文件實時寫入負載均衡器，而後負載均衡器 reload 該規則即可實現服務發現，即動態映射。

3.1.3 Nginx-Ingress工做原理

ingress controller經過和kubernetes api交互，動態的去感知集羣中ingress規則變化；而後讀取它，按照自定義的規則，規則就是寫明瞭哪一個域名對應哪一個service，生成一段nginx配置；再寫到nginx-ingress-control的pod裏，這個Ingress controller的pod裏運行着一個Nginx服務，控制器會把生成的nginx配置寫入/etc/nginx.conf文件中；而後reload一下使配置生效。以此達到域名分配置和動態更新的問題。

說明：基於nginx服務的ingress controller根據不一樣的開發公司，又分爲：

k8s社區的ingres-nginx（https://github.com/kubernetes/ingress-nginx）
nginx公司的nginx-ingress（https://github.com/nginxinc/kubernetes-ingress）

3.2.4 Ingress Controller高可用架構

做爲集羣流量接入層，Ingress Controller的高可用性顯得尤其重要，高可用性首先要解決的就是單點故障問題，通常經常使用的是採用多副本部署的方式，咱們在Kubernetes集羣中部署高可用Ingress Controller接入層一樣採用多節點部署架構，同時因爲Ingress做爲集羣流量接入口，建議採用獨佔Ingress節點的方式，以免業務應用與Ingress服務發生資源爭搶。

如上述部署架構圖，由多個獨佔Ingress實例組成統一接入層承載集羣入口流量，同時可依據後端業務流量水平擴縮容Ingress節點。固然若是您前期的集羣規模並不大，也能夠採用將Ingress服務與業務應用混部的方式，但建議進行資源限制和隔離。

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。