Prometheus K8S部署

Prometheus K8S部署

 

部署方式：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus

源碼目錄：kubernetes/cluster/addons/prometheus

服務發現：https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

部署條件

一、K8S中部署內部DNS服務

二、已有可以使用的動態PV

配置文件

下列是已經修改好的配置文件，可根據條件自行微調

• # 訪問api受權
• prometheus-rbac.yaml

• apiVersion: v1
  # 建立 ServiceAccount 授予權限
  kind: ServiceAccount
  metadata:
    name: prometheus
    namespace: kube-system
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  ---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRole
  metadata:
    name: prometheus
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile 
  rules:
    - apiGroups:
        - ""
      # 授予的權限
      resources:
        - nodes
        - nodes/metrics
        - services
        - endpoints
        - pods
      verbs:
        - get
        - list
        - watch
    - apiGroups:
        - ""
      resources:
        - configmaps
      verbs:
        - get
    - nonResourceURLs:
        - "/metrics"
      verbs:
        - get
  ---
  # 角色綁定
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: prometheus
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: prometheus
  subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: kube-system

  配置文件
• # 管理prometheus配置文件
• prometheus-configmap.yaml

• # Prometheus configuration format https://prometheus.io/docs/prometheus/latest/configuration/configuration/
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: prometheus-config
    namespace: kube-system 
    labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: EnsureExists
  data:
    # 存放prometheus配置文件
    prometheus.yml: |
      # 配置採集目標
      scrape_configs:
      - job_name: prometheus
        static_configs:
        - targets:
          # 採集自身
          - localhost:9090
      
      # 採集：Apiserver 生存指標
      # 建立的job name 名稱爲 kubernetes-apiservers
      - job_name: kubernetes-apiservers
        # 基於k8s的服務發現
        kubernetes_sd_configs:
        - role: endpoints
        # 使用通訊標記標籤
        relabel_configs:
        # 保留正則匹配標籤
        - action: keep
          # 已經包含
          regex: default;kubernetes;https
          source_labels:
          - __meta_kubernetes_namespace
          - __meta_kubernetes_service_name
          - __meta_kubernetes_endpoint_port_name
        # 使用方法爲https、默認http
        scheme: https
        tls_config:
          # promethus訪問Apiserver使用認證
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          # 跳過https認證
          insecure_skip_verify: true
        # promethus訪問Apiserver使用認證
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
   
      # 採集：Kubelet 生存指標
      - job_name: kubernetes-nodes-kubelet
        kubernetes_sd_configs:
        # 發現集羣中全部的Node
        - role: node
        relabel_configs:
        # 經過regex獲取關鍵信息
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  
      # 採集：nodes-cadvisor 信息
      - job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        # 重命名標籤
        - target_label: __metrics_path__
          replacement: /metrics/cadvisor
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  
      # 採集：service-endpoints 信息
      - job_name: kubernetes-service-endpoints
        # 選定指標
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - action: keep
          regex: true
          # 指定源標籤
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_scrape
        - action: replace
          regex: (https?)
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_scheme
          # 重命名標籤採集
          target_label: __scheme__
        - action: replace
          regex: (.+)
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_path
          target_label: __metrics_path__
        - action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          source_labels:
          - __address__
          - __meta_kubernetes_service_annotation_prometheus_io_port
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_service_name
          target_label: kubernetes_name
  
      # 採集：kubernetes-services 服務指標
      - job_name: kubernetes-services
        kubernetes_sd_configs:
        - role: service
        # 黑盒探測，探測IP與端口是否可用
        metrics_path: /probe
        params:
          module:
          - http_2xx
        relabel_configs:
        - action: keep
          regex: true
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_probe
        - source_labels:
          - __address__
          target_label: __param_target
        # 使用 blackbox進行黑盒探測
        - replacement: blackbox
          target_label: __address__
        - source_labels:
          - __param_target
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - source_labels:
          - __meta_kubernetes_service_name
          target_label: kubernetes_name
  
      # 採集： kubernetes-pods 信息
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - action: keep
          regex: true
          source_labels:
          # 只保留採集的信息
          - __meta_kubernetes_pod_annotation_prometheus_io_scrape
        - action: replace
          regex: (.+)
          source_labels:
          - __meta_kubernetes_pod_annotation_prometheus_io_path
          target_label: __metrics_path__
        - action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          source_labels:
          # 採集地址
          - __address__
          # 採集端口 
          - __meta_kubernetes_pod_annotation_prometheus_io_port
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          source_labels:
          - __meta_kubernetes_namespace
          target_label: kubernetes_namespace
        - action: replace
          source_labels:
          - __meta_kubernetes_pod_name
          target_label: kubernetes_pod_name
      alerting:
        # 告警配置文件
        alertmanagers:
        - kubernetes_sd_configs:
            # 採用動態獲取
            - role: pod
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
          - source_labels: [__meta_kubernetes_namespace]
            regex: kube-system 
            action: keep
          - source_labels: [__meta_kubernetes_pod_label_k8s_app]
            regex: alertmanager
            action: keep
          - source_labels: [__meta_kubernetes_pod_container_port_number]
            regex:
            action: drop

  配置文件
• # 將prometheus暴露訪問
• prometheus-service.yaml

• apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: prometheus
    # 部署命名空間 
    namespace: kube-system
    labels:
      k8s-app: prometheus
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
      version: v2.2.1
  spec:
    serviceName: "prometheus"
    replicas: 1
    podManagementPolicy: "Parallel"
    updateStrategy:
     type: "RollingUpdate"
    selector:
      matchLabels:
        k8s-app: prometheus
    template:
      metadata:
        labels:
          k8s-app: prometheus
        annotations:
          scheduler.alpha.kubernetes.io/critical-pod: ''
      spec:
        priorityClassName: system-cluster-critical
        serviceAccountName: prometheus
        # 初始化容器
        initContainers:
        - name: "init-chown-data"
          image: "busybox:latest"
          imagePullPolicy: "IfNotPresent"
          command: ["chown", "-R", "65534:65534", "/data"]
          volumeMounts:
          - name: prometheus-data
            mountPath: /data
            subPath: ""
        containers:
          - name: prometheus-server-configmap-reload
            image: "jimmidyson/configmap-reload:v0.1"
            imagePullPolicy: "IfNotPresent"
            args:
              - --volume-dir=/etc/config
              - --webhook-url=http://localhost:9090/-/reload
            volumeMounts:
              - name: config-volume
                mountPath: /etc/config
                readOnly: true
            resources:
              limits:
                cpu: 10m
                memory: 10Mi
              requests:
                cpu: 10m
                memory: 10Mi
  
          - name: prometheus-server
            # 主要使用鏡像
            image: "prom/prometheus:v2.2.1"
            imagePullPolicy: "IfNotPresent"
            args:
              - --config.file=/etc/config/prometheus.yml
              - --storage.tsdb.path=/data
              - --web.console.libraries=/etc/prometheus/console_libraries
              - --web.console.templates=/etc/prometheus/consoles
              - --web.enable-lifecycle
            ports:
              - containerPort: 9090
            readinessProbe:
              # 健康檢查
              httpGet:
                path: /-/ready
                port: 9090
              initialDelaySeconds: 30
              timeoutSeconds: 30
            livenessProbe:
              httpGet:
                path: /-/healthy
                port: 9090
              initialDelaySeconds: 30
              timeoutSeconds: 30
            # based on 10 running nodes with 30 pods each
            resources:
              limits:
                cpu: 200m
                memory: 1000Mi
              requests:
                cpu: 200m
                memory: 1000Mi
            # 數據卷
            volumeMounts:
              - name: config-volume
                mountPath: /etc/config
              - name: prometheus-data
                mountPath: /data
                subPath: ""
        terminationGracePeriodSeconds: 300
        volumes:
          - name: config-volume
            configMap:
              name: prometheus-config
    volumeClaimTemplates:
    - metadata:
        name: prometheus-data
      spec:
        # 使用動態PV、修改成已建立的PV動態存儲
        storageClassName: managed-nfs-storage
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: "16Gi"

  配置文件
• # 經過有狀態的形式將prometheus部署
• prometheus-statefulset.yaml

• kind: Service
  apiVersion: v1
  metadata:
    name: prometheus
    # 指定命名空間
    namespace: kube-system
    labels:
      kubernetes.io/name: "Prometheus"
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  spec:
    # 添加外部訪問
    type: NodePort
    # 指定內部訪問協議
    ports:
      - name: http
        port: 9090
        protocol: TCP
        targetPort: 9090
    selector:
      k8s-app: prometheus

  配置文件 

部署

一、下載github包：https://github.com/kubernetes/kubernetes/

二、複製文件到指定目錄

mkdir ~/prometheus
cp ~/kubernetes/cluster/addons/prometheus/* ~/prometheus/

三、進入到目錄

cd ~/prometheus/

四、k8s經過配置文件建立運行容器

kubectl apply -f prometheus-rbac.yaml
kubectl apply -f prometheus-configmap.yaml
kubectl apply -f prometheus-statefulset.yaml
kubectl apply -f prometheus-service.yaml 

五、查看建立資源

kubectl get pod,svc -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
pod/coredns-64479cf49b-lsqqn   1/1     Running   0          75m
pod/prometheus-0               2/2     Running   0          2m12s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns     ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP,9153/TCP   75m
service/prometheus   NodePort    10.0.0.170   <none>        9090:42575/TCP           8s

六、測試經過端口開啓端口訪問監控端