promethues系列之:追蹤k8s容器指標的打tag的流程

k8s中容器資源的監控

在promethues中如何配置採集容器指標

• 採用promethues的kubernetes_sd_configs中 node級別的role

- job_name: kubernetes-nodes-cadvisor
  honor_timestamps: false
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - role: node
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - separator: ;
    regex: (.*)
    target_label: __metrics_path__
    replacement: /metrics/cadvisor
    action: replace

cadvisor架構圖

cadvisor中 POD

• 在查看cadvisor代碼時發現有一種container_name=="POD"的容器，查了下是 k8s中的pause pod

下面追蹤下打tag的過程:以pod cpu使用率爲例

kubelet最終tag

• 對應cadvisor指標爲 container_cpu_usage_seconds_total,能夠看到最終查詢出來的有以下tag
• 那咱們會好奇標識app或service的tag:pod，pod_name，container，container_name是如何打上去的呢

訪問集成在kubelet中的cadvisor的tag

• curl localhost:4194/metrics

• 能夠發現除了cpu是container_cpu_usage_seconds_total指標特有的tag以外,還有id,name,namespace,pod_name,container_name,image這幾個tag
• 上述tag做爲cadvisor通用tag會附加在每個metric上面
• 其實在裸的cadvisor中只有id,image,name三個tag原始cadvisor 打tag
• namespace,pod_name,container_name等屬性是k8s纔有的,cadvisor確定沒法感知pod信息,說明是k8s注入的

kubelet內置的cadvisor中使用了自定義的PrometheusLabelsFunc

以k8s 1.15版本爲例
代碼在：E:go_pathsrcgithub.comkuberneteskubernetespkgkubeletserverserver.go

func containerPrometheusLabelsFunc(s stats.Provider) metrics.ContainerLabelsFunc {
    // containerPrometheusLabels maps cAdvisor labels to prometheus labels.
    return func(c *cadvisorapi.ContainerInfo) map[string]string {
        // Prometheus requires that all metrics in the same family have the same labels,
        // so we arrange to supply blank strings for missing labels
        var name, image, podName, namespace, containerName string
        if len(c.Aliases) > 0 {
            name = c.Aliases[0]
        }
        image = c.Spec.Image
        if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNameLabel]; ok {
            podName = v
        }
        if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNamespaceLabel]; ok {
            namespace = v
        }
        if v, ok := c.Spec.Labels[kubelettypes.KubernetesContainerNameLabel]; ok {
            containerName = v
        }
        // Associate pod cgroup with pod so we have an accurate accounting of sandbox
        if podName == "" && namespace == "" {
            if pod, found := s.GetPodByCgroupfs(c.Name); found {
                podName = pod.Name
                namespace = pod.Namespace
            }
        }
        set := map[string]string{
            metrics.LabelID:    c.Name,
            metrics.LabelName:  name,
            metrics.LabelImage: image,
            "pod_name":         podName,
            "pod":              podName,
            "namespace":        namespace,
            "container_name":   containerName,
            "container":        containerName,
        }
        return set
    }
}

k8s 1.15 1.16版本對於pod和pod_name的變化

• 通過觀察發現1.15的pod pod_name都有，1.16只有pod
• 這是由於 在k8s 1.16版本爲了統一cadvisor和kube-stats指標tag作了變動[這個pr] (https://github.com/kubernetes...

kubelet啓動時能夠使用--node-labels注入node級別tag

--node-labels=os.name=xxxx,os.version=xxxx,os.architecture=amd64,

• 這些tag會轉化爲promethues metric命名方式xxx_xxx
• 最後追加爲os_version=xxx,os_architecture=amd64