traefik Ingress https配置

環境

. kubernetes 1.14.3

. traefik V1.7.12

.IP 192.168.30.35

.kubectl label nodes ingress ingress=yes

https證書申請

推薦使用acme.sh 申請免費證書，具體方法不作詳細介紹
使用自簽證書

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=*.mddgame.com"

traefik配置

添加traefik.toml文件：

一、http,https 同時對外提供服務
defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/certs/tls.crt"
      keyFile = "/certs/tls.key"

二、http 強制跳轉https
defaultEntryPoints = ["http","https"]
[kubernetes]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      CertFile = "/certs/tls.crt"
      KeyFile = "/certs/tls.key"

其中tls.crt和tls.key就是證書文件，注意證書文件名必須爲固定。掛載到容器內後就會讀到該文件。

建立證書secret 方便掛在
kubectl -n kube-system create secret generic mddgame-tls-cert --from-file=tls.key --from-file=tls.crt
注意：因爲secret是不能跨命名空間的，若是應用是部署在default命名空間或者其它命名空間，那還須要在default命名空間建立一個該secret，修改上面最後面的-n kube-system爲其它命名空間的名字便可。
建立configmap
kubectl create configmap traefik-conf --from-file=traefik.toml -n kube-system

traefik部署配置

一、traefik-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
    - extensions
    resources:
    - ingresses/status
    verbs:
    - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik
subjects:
- kind: ServiceAccount
  name: traefik
  namespace: kube-system

2 建立 traefik 用戶

vi traefik-serviceaccount.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik
  namespace: kube-system

三、deployment 方式部署yaml

vi traefik-deployment-https.yaml
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik
  namespace: kube-system
  labels:
    k8s-app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik
  template:
    metadata:
      labels:
        k8s-app: traefik
        name: traefik
    spec:
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
      volumes:
      - name: ssl
        secret:
          secretName: mddgame-tls-cert
      - name: config
        configMap:
          name: traefik-conf
          defaultMode: 0644
          items:
          - key: traefik.toml
            path: traefik.toml 
      hostNetwork: true       # 若是不使用hostNetwork 配置hostPort 443端口映射到宿主機會出現訪問不了k8s api 10.64.0.1:443 端口 
      dnsPolicy: ClusterFirstWithHostNet      
      containers:
      - image: traefik
        name: traefik
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /certs         
          name: "ssl"
        - mountPath: /etc/traefik.toml
          subPath: traefik.toml
          name: "config"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: admin
          containerPort: 8080
        args:
        - --api
        - --web
        - --api.dashboard
        - --logLevel=INFO
        - --web.metrics
        - --metrics.prometheus
        - --web.metrics.prometheus
        - --kubernetes
        - --traefiklog
        - --traefiklog.format=json
        - --accesslog
        - --accesslog.format=json
        - --accessLog.fields.headers.defaultMode=redact
        - --insecureskipverify=true
        - --configFile=/etc/traefik.toml       
      nodeSelector:
        ingress: "yes"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/ingress
        operator: Equal

四、daemonset 方式部署

vi traefik-daemonset-https.yaml
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: traefik
  namespace: kube-system
  labels:
    k8s-app: traefik
spec:
  selector:
    matchLabels:
      k8s-app: traefik
  template:
    metadata:
      labels:
        k8s-app: traefik
        name: traefik
    spec:
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
      volumes:
      - name: ssl
        secret:
          secretName: mddgame-tls-cert
      - name: config
        configMap:
          name: traefik-conf
          defaultMode: 0644
          items:
          - key: traefik.toml
            path: traefik.toml
      hostNetwork: true #若是不使用hostNetwork 配置hostPort 443端口映射到宿主機會出現訪問不了k8s api 10.64.0.1:443 端口 
      dnsPolicy: ClusterFirstWithHostNet      
      containers:
      - image: traefik
        name: traefik
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /certs         
          name: "ssl"
        - mountPath: /etc/traefik.toml
          subPath: traefik.toml
          name: "config"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: admin
          containerPort: 8080
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --api
        - --web
        - --api.dashboard
        - --logLevel=INFO
        - --web.metrics
        - --metrics.prometheus
        - --web.metrics.prometheus
        - --kubernetes
        - --traefiklog
        - --traefiklog.format=json
        - --accesslog
        - --accesslog.format=json
        - --accessLog.fields.headers.defaultMode=redact
        - --insecureskipverify=true
        - --configFile=/etc/traefik.toml       
      nodeSelector:
        ingress: "yes"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/ingress
        operator: Equal

五、建立traefik Service

vi traefik-service.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: traefik
  name: traefik
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik
  clusterIP: None
  ports:
    - protocol: TCP
      port: 80
      name: http
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin

六、建立traefik ServiceMonitor

vi traefik-serviceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: traefik
  name: traefik
  namespace: monitoring
spec:
  endpoints:
  - honorLabels: true
    interval: 15s
    port: admin
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: traefik

七、建立 traefik dashboard Ingress

vi traefik-dashboard.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-dashboard
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/frontend-entry-points: http,https
spec:
  rules:
  - host: traefik.mddgame.com
    http:
      paths:
        - backend:
            serviceName: traefik
            servicePort: 8080
  tls:
   - secretName: mddgame-tls-cert

執行yaml 建立traefik 服務

kubectl apply -f .

驗證traefik 服務

kubectl get all -n kube-system | grep traefik
root@Qist:/mnt/e/work/k8s-game# kubectl get all -n kube-system | grep traefik

pod/traefik-2d5k8                          1/1     Running   0          16h

service/traefik                ClusterIP   None            <none>        80/TCP,443/TCP,8080/TCP   16h

daemonset.apps/traefik       1         1         1       1            1           ingress=yes     16h
寫hosts
192.168.30.35 traefik.mddgame.com
訪問traefik.mddgame.com

http 訪問

https 訪問

建立測試

vi nginx.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: nginx
  template:
    metadata:
      labels:
        k8s-app: nginx
    spec:
      containers:
        - name: nginx
          image: docker.mddgame.com/library/nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
              name: web
              protocol: TCP
            - containerPort: 8080
              name: vts
              protocol: TCP
          readinessProbe:
            failureThreshold: 10
            httpGet:
              path: /healthz
              port: vts
              scheme: HTTP
            initialDelaySeconds: 3
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
        - name: nginx-vts-exporter
          image: docker.mddgame.com/library/nginx-vts-exporter
          imagePullPolicy: IfNotPresent
          args:
          - "-nginx.scrape_uri=http://localhost:8080/format/json"
          ports:
            - containerPort: 9913
              name: http-metrics
              protocol: TCP
          resources:
            requests:
              memory: 30Mi
              cpu: 102m
            limits:
              memory: 50Mi
              cpu: 250m
---
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: nginx
  name: nginx
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/affinity: "true" # 後端基於Cookie會話
    traefik.ingress.kubernetes.io/load-balancer-method: drr   #修改負載方式
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  selector:
    k8s-app: nginx
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: vts
    - protocol: TCP
      port: 9913
      name: http-metrics
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
    traefik.ingress.kubernetes.io/frontend-entry-points: http,https
spec:
  rules:
  - host: nginx.mddgame.com
    http:
      paths:
        - path: /
          backend:
            serviceName: nginx
            servicePort: 80
  tls:
   - secretName: mddgame-tls-cert
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: nginx
  name: nginx
spec:
  endpoints:
  - honorLabels: true
    interval: 15s
    port: http-metrics
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: nginx

建立nginx 服務
kubectl apply -f .
hosts 寫入
192.168.30.35 nginx.mddgame.com

正常打開

外部業務使用traefik 對外提供服務

以Confluence 爲例
vi wiki.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: wiki
  name: wiki
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/affinity: "true"
    traefik.ingress.kubernetes.io/load-balancer-method: drr
spec:
  clusterIP: None
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    k8s-app: wiki
  name: wiki
  namespace: default
subsets:
- addresses:
  - ip: 192.168.30.11 # 多個ip 能夠直接在下一行添加 -ip:192.168.30.22 
  ports:
  - name: http
    port: 8080
    protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: wiki
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
    traefik.ingress.kubernetes.io/frontend-entry-points: http,https
    traefik.ingress.kubernetes.io/redirect-entry-point: https # 強制調整到https
spec:
  rules:
  - host: wiki.mddgame.com
    http:
      paths:
        - path: /
          backend:
            serviceName: wiki
            servicePort: 8080
  tls:
   - secretName: mddgame-tls-cert

綁定hosts
192.168.30.35 wiki.mddgame.com
驗證是否能打開

可以正常打開