Kubernetes 1.9集羣使用traefik發佈服務

在前文中介紹了在kubernetes 1.5.2集羣環境中使用traefik進行服務發佈。Traefik採用daemonset方式部署，鏈接api-server走的是http協議，也未配置rbac。本文將介紹在k8s 1.9版本中使用deployment方式部署traefik來進行服務發佈。

在開始以前，須要先了解一下什麼是RBAC。RBAC（基於角色的訪問控制）使用 rbac.authorization.k8s.io  API 組來實現權限控制，RBAC 容許管理員經過 Kubernetes API 動態的配置權限策略。在 1.6 版本中 RBAC 還處於 Beat 階段，若是想要開啓 RBAC 受權模式須要在 apiserver 組件中指定 --authorization-mode=RBAC 選項。

在 RBAC API 的四個重要概念：
Role：是一系列的權限的集合，例如一個角色能夠包含讀取 Pod 的權限和列出 Pod 的權限
ClusterRole： 跟 Role 相似，可是能夠在集羣中處處使用（ Role 是 namespace 一級的）
RoloBinding：把角色映射到用戶，從而讓這些用戶繼承角色在 namespace 中的權限。
ClusterRoleBinding： 讓用戶繼承 ClusterRole 在整個集羣中的權限。

簡單點說RBAC實現了在k8s集羣中對api-server的鑑權，更多的RBAC知識點請查閱官方文檔：https://kubernetes.io/docs/admin/authorization/rbac/

1、給集羣的節點打上label
由於選擇deployment方式部署，因此要給集羣的節點打上label，後續選擇nodeSelector指定traefik=proxy，副本數和集羣節點數一致的時候，全部的節點上都會運行一個pod

# kubectl get nodes --show-labels
# kubectl label  node vm1 traefik=proxy
# kubectl label  node vm2 traefik=proxy
# kubectl get nodes --show-labels

2、準備yaml文件
一、rbac文件

# cat traefik-rbac.yaml 
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

在啓用rbac的環境下，若是鑑權未配置清楚，則traefik pod會報錯以下

E0226 00:15:27.729832 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope 
E0226 00:15:29.013298 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:default" cannot list endpoints at the cluster scope 
E0226 00:15:29.213354 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:kube-system:default" cannot list secrets at the cluster scope 
E0226 00:15:29.698574 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:kube-system:default" cannot list ingresses.extensions at the cluster scope 
E0226 00:15:30.411837 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope 
E0226 00:15:31.912887 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:default" cannot list endpoints at the cluster scope

二、traefik的deployment文件

# cat traefik-deployment.yaml   
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      hostNetwork: true
      nodeSelector:
        traefik: proxy
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik
        name: traefik-ingress-lb
        ports:
        - name: web
          containerPort: 80
          hostPort: 80
        - name: admin
          containerPort: 8081
        args:
        - --web
        - --web.address=:8081
        - --kubernetes

三、traefik的service文件

# cat traefik-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 80
    targetPort: 8081

四、經過yaml文件建立clusterrole、clusterrolebinding、deployment、serviceaccount、service

# ls
# kubectl create -f traefik-rbac.yaml 
# kubectl create -f traefik-deployment.yaml 
# kubectl create -f traefik-service.yaml

# kubectl get pod -n kube-system
# kubectl get svc -n kube-system
# kubectl get svc

能夠看到集羣中default namespace中存在一個frontend服務。kube-system namespace中存在nginx-test、traefik-web-ui、kubernetes-dashboard三個服務。咱們後續將建立4個ingress

經過web-ui能夠看到在兩個節點上各運行了一個pod

3、經過yaml文件建立ingress

# cat ui.yaml 
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 80
    targetPort: 8081
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: traefik-ui
    http:
      paths:
      - backend:
          serviceName: traefik-web-ui
          servicePort: 80

# cat webui-ing.yaml                  
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: k8s.webui
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard 
          servicePort: 443

# cat redis-ing.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: k8s.frontend
    http:
      paths:
      - backend:
          serviceName: frontend 
          servicePort: 80

# cat nginx-ing.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-nginx-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: test.fjhb.cn
    http:
      paths:
      - backend:
          serviceName: nginx-test
          servicePort: 80

# kubectl create -f ui.yaml 
# kubectl create -f webui-ing.yaml 
# kubectl create -f redis-ing.yaml 
# kubectl get ingress 
# kubectl get ingress -n kube-system

3、驗證
一、經過訪問traefik service對應的nodeport端口，4個ingress配置都加載到了

二、修改測試機hosts文件，將4個域名的解析分配到兩臺節點上

三、瀏覽器訪問測試

這裏出現500錯誤的緣由是，後端的kubernetes-dashboard配置的是https協議



能夠在health頁面看到http狀態碼的統計信息