Kubernetes 部署 Nginx Ingress Controller 之 nginxinc/kubernetes-ingress

更新：這裏用的是 nginxinc/kubernetes-ingress ，還有個 kubernetes/ingress-nginx ，它們的區別見 Differences Between nginxinc/kubernetes-ingress and kubernetes/ingress-nginx Ingress Controllers ，後來咱們選用了 kubernetes/ingress-nginx ，詳見博文。

開始天真地覺得只要寫一個 ingress 配置文件並部署好就好了。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cnblogs-ingress
spec:
  rules:
  - host: q.cnblogs.com
    http:
      paths:
        - backend:
            serviceName: q-web
            servicePort: 80

# kubectl apply -f cnblogs-ingress.yaml
# kubectl get ingress
NAME              HOSTS           ADDRESS   PORTS   AGE
cnblogs-ingress   q.cnblogs.com             80      6h18

但部署後發現全部 node 服務器上沒有任何進程監聽 80 端口，顯然不對。

從 k8s 幫助文檔中知道了答案：

You must have an ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.
In order for the Ingress resource to work, the cluster must have an ingress controller running.
Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster. Use this page to choose the ingress controller implementation that best fits your cluster.

原來 k8s 沒有內置 ingress controller ，須要安裝第三方的 ingress controller ，好比 nginx ingress controller ，上面經過 cnblogs-ingress.yaml 只是建立了 ingress 資源。那爲何經過 deployment.yaml 建立了 deployment 資源就能正常部署 pod ？那是由於 kube-controller-manager 中內置了 deployment controller 。

咱們選用 nginx ingress controller ，部署操做步驟以下（參考文檔）：
1）從 github 上籤出 kubernetes-ingress 倉庫

$ git clone https://github.com/nginxinc/kubernetes-ingress/
$ cd kubernetes-ingress
$ git checkout v1.6.1 -f
$ cd deployments

2）建立 namespace 與 ServiceAccount ，都叫 nginx-ingress

kubectl apply -f common/ns-and-sa.yaml

3）建立 cluster role 與 cluster role binding

kubectl apply -f rbac/rbac.yaml

4）建立 secret
使用本身的證書文件建立 secret

kubectl create secret tls default-server-secret --cert=path/to/cert.pem --key=path/to/key.pem

或者使用 nginx-ingress 自帶的證書建立 sescret

kubectl apply -f common/default-server-secret.yaml

5）建立 ConfigMap

kubectl apply -f common/nginx-config.yaml

6）建立 custom resource definitions

kubectl apply -f common/custom-resource-definitions.yaml

7）建立 DaemonSet

kubectl apply -f daemon-set/nginx-ingress.yaml

8）查看 pod 是否部署成功

$ kubectl get pods --namespace=nginx-ingress                                                                                                1 ↵
NAME                  READY   STATUS    RESTARTS   AGE
nginx-ingress-7xdzp   1/1     Running   5          12m
nginx-ingress-rs4th   1/1     Running   0          114s
nginx-ingress-w2fnh   1/1     Running   0          12m
nginx-ingress-z54r6   1/1     Running   5          12m

9）建立監聽 31080 端口的 NodePort 類型的 service

配置文件 nodeport.yaml （去掉了443端口）
注0：nodePort 只能使用 30000-32767 範圍的端口。
注1：去掉了443端口，咱們在最前端使用了阿里雲負載均衡，請求都經過 http 轉發。

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  type: NodePort 
  ports:
  - nodePort: 31080
     port: 80
     targetPort: 80
     protocol: TCP
     name: http
  selector:
    app: nginx-ingress

部署命令

kubectl apply -f service/nodeport.yaml

10）檢查 nginx-ingress 部署成功

進入 nginx-ingress 容器

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress /bin/bash

查看 nginx 配置

cat /etc/nginx/conf.d/production-cnblogs-ingress.conf

確認 ingress 中添加的轉發規則已被導入

upstream production-cnblogs-ingress-q.cnblogs.com-q-web-80 {
    zone production-cnblogs-ingress-q.cnblogs.com-q-web-80 256k;
    random two least_conn;
    
    server 192.168.107.211:80 max_fails=1 fail_timeout=10s max_conns=0;
    server 192.168.186.72:80 max_fails=1 fail_timeout=10s max_conns=0;  
}

server {
    listen 80;
    server_tokens on;
    server_name q.cnblogs.com;
    
    location / {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        
        proxy_pass http://production-cnblogs-ingress-q.cnblogs.com-q-web-80;
    }
}

至此 nginx-ingress 部署成功。

解決轉發 X-Forwarded-Proto 請求頭問題

解決方法：在 ingress 配置文件中添加 nginx.org/redirect-to-https: "true"，詳見博問 K8s Nginx Ingress Controller 轉發 X-Forwarded-Proto 請求頭的問題

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/redirect-to-https: "true"

添加 proxy_set_header 配置

經過 ingress 的 nginx.org/location-snippets 註解添加以下的配置：

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    nginx.org/location-snippets: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

七層負載均衡對 nginx-ingress 進行健康檢查問題

詳見博問 阿里雲負載均衡對 K8s Nginx Ingress 的健康檢查問題

基於二級域名自動轉發到 service 的實現方法

詳見博問 K8s Ingress 如何自動根據主機名中的二級域名匹配 service

應用獲取不到客戶端真實 IP 地址的問題

詳見博問 K8s 中 ASP.NET Core 應用獲取不到客戶端真實 IP 地址