nginx-ingress-controller日誌持久化方案

時間  2020-03-31
標籤 nginx ingress controller 日誌 持久 方案 欄目 Nginx 简体版
原文   原文鏈接

最近看到一篇公衆號講了nginx-ingress-controller的應用。下面有人評論如何作日誌持久化，恰好工做上遇到該問題，整理一個方案，僅供參考。node

nginx-ingress-controller的日誌

nginx-ingress-controller的日誌包括三個部分：nginx

  • controller日誌： 輸出到stdout，經過啓動參數中的–log_dir可已配置輸出到文件，重定向到文件後會自動輪轉，但不會自動清理
  • accesslog：輸出到stdout，經過nginx-configuration中的字段能夠配置輸出到哪一個文件。輸出到文件後不會自動輪轉或清理
  • errorlog：輸出到stderr，配置方式與accesslog相似。

給controller日誌落盤

  1. 給nginx-ingress-controller掛一個hostpath： /data/log/nginx_ingress_controller/ 映射到容器裏的/var/log/nginx_ingress_controller/ ，
  2. 給nginx-ingress-controller配置log-dir和logtostderr參數，將日誌重定向到/var/log/nginx_ingress_controller/中。

controller的日誌須要作定時清理。因爲controller的日誌是經過klog(k8s.io/klog)輸出的，會進行日誌滾動，因此咱們經過腳本定時清理必定時間以前的日誌文件便可。docker

給nginx日誌落盤

  1. 修改configmap： nginx-configuration。配置accesslog和errorlog的輸出路徑，替換默認的stdout和stderr。輸出路徑咱們能夠與controller一致，便於查找。

  2. accesslog和errorlog都只有一個日誌文件，咱們可使用logrotate進行日誌輪轉，將輸出到宿主機上的日誌進行輪轉和清理。配置如：api

    $ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
    su root list
    rotate 7
    daily
    maxsize 50M
    copytruncate
    missingok
    create 0644 www-data root
}
  3. 官方提供的模板中，nginx-ingress-controller默認都是以33這個用戶登陸啓動容器的，所以掛載hostpath路徑時存在權限問題。咱們須要手動在機器上執行chown -R 33:33 /data/log/nginx_ingress_controller.

自動化ops

nginx日誌落盤中，第二、3兩點均須要人工運維，有什麼解決辦法嗎？bash

問題的關鍵是：有什麼辦法能夠在nginx-ingress-controller容器啓動以前加一個hook，將宿主機的指定目錄執行chown呢？app

能夠用initContainer。initcontainer必須在containers中的容器運行前運行完畢併成功退出。利用這一k8s特性，咱們開發一個docker image，裏面只執行以下腳本：運維

#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo "try to set dir: $logdir 's group as $userID"
chown  -R  $userID:$userID $logdir

腳本讀取一些環境變量， 確認須要修改哪一個目錄，改爲怎樣的user group。tcp

將腳本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，做爲initcontainers。 注意要對該initcontainer配置環境變量和volumeMount.ui

再說第二點，咱們注意到nginx-ingress-controller的基礎鏡像中就自帶了logrotate，那麼問題就簡單了，咱們將寫好的logrotate配置文件以configmap的形式掛載到容器中就能夠了。spa

一個deploy yaml以下：

---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default
  namespace: kube-system
spec:
  backend:
    serviceName: default-http-backend
    servicePort: 80
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: kube-system
  labels:
    app: ingress-nginx
data:
  use-forwarded-headers: "true"
  # 此處配置nginx日誌的重定向目標
  access-log-path: /var/log/nginx_ingress_controller/access.log
  error-log-path: /var/log/nginx_ingress_controller/error.log

---

# 建立一個configmap，配置nginx日誌的輪轉策略，對應的是nginx日誌在容器內的日誌文件
apiVersion: v1
data:
  nginx.log: |
    {{ user_nginx_log.host_path }}/access.log {
        rotate {{ user_nginx_log.rotate_count }}
        daily
        maxsize {{ user_nginx_log.rotate_size }}
        minsize 10M
        copytruncate
        missingok
        create 0644 root root
    }
    {{ user_nginx_log.host_path }}/error.log {
        rotate {{ user_nginx_log.rotate_count }}
        daily
        maxsize {{ user_nginx_log.rotate_size }}
        minsize 10M
        copytruncate
        missingok
        create 0644 root root
    }
kind: ConfigMap
metadata:
  name: nginx-ingress-logrotate
  namespace: kube-system
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      tolerations:
      - key: dedicated
        value: ingress-nginx
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "system/ingress"
                operator: In
                values:
                - "true"
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      # 配置initcontainer，確保在nginx-ingress-controller容器啓動前將日誌目錄的權限配置好
      initContainers:
      - name: adddirperm
        image: "{{ image_registry.addr }}/{{ image.adddirperm }}"
        env:
        - name: LOG_DIR
          value: /var/log/nginx_ingress_controller
        - name: USER_ID
           value: "33"
        volumeMounts:
        - name: logdir
          mountPath: /var/log/nginx_ingress_controller
      containers:
      - name: nginx-ingress-controller
        image: "{{ image_registry.addr }}/{{ image.ingress }}"
        imagePullPolicy: IfNotPresent
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        
        # 設置controller日誌的輸出路徑和方式
        - --log_dir=/var/log/nginx_ingress_controller
        - --logtostderr=false
        securityContext:
          capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
          # www-data -> 33
          runAsUser: 33
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        volumeMounts:
        # 配置掛載容器中控制器組件和nginx的日誌輸出路徑
        - name: logdir
          mountPath: /var/log/nginx_ingress_controller
        # 配置nginx日誌的logrotate配置掛載路徑
        - name: logrotateconf
          mountPath: /etc/logrotate.d/nginx.log
          subPath: nginx.log
      volumes:
      # 控制器組件和nginx的日誌輸出路徑爲宿主機的hostpath
      - name: logdir
        hostPath:
          path: {{ user_nginx_log.host_path }}
          type: ""
      # nginx日誌的輪轉配置文件來自於configmap
      - name: logrotateconf
        configMap:
          name: nginx-ingress-logrotate
          items:
          - key: nginx.log
            path: nginx.log
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    app: default-http-backend
spec:
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      tolerations:
      - key: dedicated
        value: ingress-nginx
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "system/ingress"
                operator: In
                values:
                - "true"
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: "{{ image_registry.addr }}/{{ image.http_backend }}"
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

最後，有的人建議將initcontainer去掉，改成基於原有的nginx-ingress-controller鏡像加一層layer，將配置路徑權限的腳本放在該層執行。 我的認爲這種方法既不美觀，也不方便。惟一的好處僅在於deploy yaml仍然簡潔（但少不了volumeMount之類的配置）。不過仍是看我的使用感覺吧~

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程