log-pilot：k8s中日誌收集神器

k8s中pod的日誌收集有2種常見的解決方案；

• 方案一：使用fluentd做爲daemonset收集stdout和/var/lib/containers目錄下的全部日誌（由於對fluentd不太熟悉，因此感受麻煩）；

• 方案二：使用filebeat做爲sidecar方式（這種方式過於繁瑣，須要在每一個pod中添加這個容器）

無心間發現阿里雲開源的log-pilot收集k8s的日誌真的超級方便，配置也簡單；

官方介紹：

github地址：https://github.com/AliyunContainerService/log-pilot

log-pilot官方介紹：https://yq.aliyun.com/articles/674327

log-pilot官方搭建：https://yq.aliyun.com/articles/674361?spm=a2c4e.11153940.0.0.21ae21c3mTKwWS

log-pilot的daemonset文件：

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: log-pilot
  labels:
    app: log-pilot
  # 設置指望部署的namespace
  namespace: kube-system
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: log-pilot
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      # 是否容許部署到Master節點上
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: log-pilot
        # 版本請參考https://github.com/AliyunContainerService/log-pilot/releases
        image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.7-filebeat
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 200m
            memory: 200Mi
        env:
          - name: "NODE_NAME"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "LOGGING_OUTPUT"
            value: "elasticsearch"
          # 請確保集羣到ES網絡可達
          - name: "ELASTICSEARCH_HOSTS"
            value: "10.10.5.78:9200"
          # 配置ES訪問權限
          #- name: "ELASTICSEARCH_USER"
          #  value: "{es_username}"
          #- name: "ELASTICSEARCH_PASSWORD"
          #  value: "{es_password}"
        volumeMounts:
        - name: sock
          mountPath: /var/run/docker.sock
        - name: root
          mountPath: /host
          readOnly: true
        - name: varlib
          mountPath: /var/lib/filebeat
        - name: varlog
          mountPath: /var/log/filebeat
        - name: localtime
          mountPath: /etc/localtime
          readOnly: true
        livenessProbe:
          failureThreshold: 3
          exec:
            command:
            - /pilot/healthz
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
      terminationGracePeriodSeconds: 30
      volumes:
      - name: sock
        hostPath:
          path: /var/run/docker.sock
      - name: root
        hostPath:
          path: /
      - name: varlib
        hostPath:
          path: /var/lib/filebeat
          type: DirectoryOrCreate
      - name: varlog
        hostPath:
          path: /var/log/filebeat
          type: DirectoryOrCreate
      - name: localtime
        hostPath:
          path: /etc/localtime

建立nginx測試pod收集日誌示例：

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: node-affinity
spec:
  selector:
    matchLabels:
      app: node-affinity
  replicas: 3
  template:
    metadata:
      labels:
        app: node-affinity
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        env:
        - name: aliyun_logs_nginx
          value: "stdout"
---
apiVersion: v1
kind: Service
metadata:
  name: node-affinity
spec:
  selector:
    app: node-affinity
  ports:
  - port: 80
    targetPort: 80
  type: NodePort

建立tomcat測試pod收集日誌示例：

apiVersion: v1
kind: Pod
metadata:
  name: tomcat
spec:
  containers:
  - name: tomcat
    image: "tomcat:8.0"
    env:
    # 一、stdout爲約定關鍵字，表示採集標準輸出日誌
    # 二、配置標準輸出日誌採集到ES的catalina索引下
    - name: aliyun_logs_catalina
      value: "stdout"
    # 一、配置採集容器內文件日誌，支持通配符
    # 二、配置該日誌採集到ES的access索引下
    - name: aliyun_logs_access
      value: "/usr/local/tomcat/logs/catalina.*.log"
    # 容器內文件日誌路徑須要配置emptyDir
    volumeMounts:
      - name: tomcat-log
        mountPath: /usr/local/tomcat/logs
  volumes:
    - name: tomcat-log
      emptyDir: {}