K8s中Pod健康檢查源代碼分析

時間  2019-11-19
標籤 k8s pod 健康檢查 源代碼 分析 简体版
原文   原文鏈接

瞭解k8s中的Liveness和Readiness

Liveness: 
代表是否容器正在運行。若是liveness探測爲fail，則kubelet會kill掉容器，而且會觸發restart設置的策略。默認不設置的狀況下，該狀態爲success.
Readiness: 
代表容器是否能夠接受服務請求。若是readiness探測失敗，則endpoints控制器會從endpoints中摘除該Pod IP。在初始化延遲探測時間以前，默認是Failure。若是沒有設置readiness探測，該狀態爲success。shell

代碼分析

基於Kubernetes 1.11.0json

1.啓動探測

在kubelet啓動是時候會啓動健康檢查的探測：
kubelet.go中Run方法緩存

...
kl.probeManager.Start() //啓動探測服務
...

2.看一下probeManager都作了哪些事情

prober_manager.go中咱們看一下這段代碼：app

// Manager manages pod probing. It creates a probe "worker" for every container that specifies a
// probe (AddPod). The worker periodically probes its assigned container and caches the results. The
// manager use the cached probe results to set the appropriate Ready state in the PodStatus when
// requested (UpdatePodStatus). Updating probe parameters is not currently supported.
// TODO: Move liveness probing out of the runtime, to here.
type Manager interface {
    // AddPod creates new probe workers for every container probe. This should be called for every
    // pod created.
    AddPod(pod *v1.Pod)

    // RemovePod handles cleaning up the removed pod state, including terminating probe workers and
    // deleting cached results.
    RemovePod(pod *v1.Pod)

    // CleanupPods handles cleaning up pods which should no longer be running.
    // It takes a list of "active pods" which should not be cleaned up.
    CleanupPods(activePods []*v1.Pod)

    // UpdatePodStatus modifies the given PodStatus with the appropriate Ready state for each
    // container based on container running status, cached probe results and worker states.
    UpdatePodStatus(types.UID, *v1.PodStatus)

    // Start starts the Manager sync loops.
    Start()
}

這是一個Manager的接口聲明，該Manager負載pod的探測。當執行AddPod時，會爲Pod中每個容器建立一個執行探測任務的worker, 該worker會對所分配的容器進行週期性的探測，並把探測結果緩存。當UpdatePodStatus方法執行時，該manager會使用探測的緩存結果設置PodStatus爲近似Ready的狀態：socket

3.一「探」究竟

先看一下探測的struct

type Probe struct {
    // The action taken to determine the health of a container
    Handler `json:",inline" protobuf:"bytes,1,opt,name=handler"`
    // Number of seconds after the container has started before liveness probes are initiated.
    // More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
    // +optional
    InitialDelaySeconds int32 `json:"initialDelaySeconds,omitempty" protobuf:"varint,2,opt,name=initialDelaySeconds"`
    // Number of seconds after which the probe times out.
    // Defaults to 1 second. Minimum value is 1.
    // More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
    // +optional
    TimeoutSeconds int32 `json:"timeoutSeconds,omitempty" protobuf:"varint,3,opt,name=timeoutSeconds"`
    // How often (in seconds) to perform the probe.
    // Default to 10 seconds. Minimum value is 1.
    // +optional
    PeriodSeconds int32 `json:"periodSeconds,omitempty" protobuf:"varint,4,opt,name=periodSeconds"`
    // Minimum consecutive successes for the probe to be considered successful after having failed.
    // Defaults to 1. Must be 1 for liveness. Minimum value is 1.
    // +optional
    SuccessThreshold int32 `json:"successThreshold,omitempty" protobuf:"varint,5,opt,name=successThreshold"`
    // Minimum consecutive failures for the probe to be considered failed after having succeeded.
    // Defaults to 3. Minimum value is 1.
    // +optional
    FailureThreshold int32 `json:"failureThreshold,omitempty" protobuf:"varint,6,opt,name=failureThreshold"`
}

initialDelaySeconds: 表示容器啓動以後延遲多久進行liveness探測
timeoutSeconds：每次執行探測的超時時間
periodSeconds：探測的週期時間
successThreshold：最少連續幾回探測成功的次數，知足該次數則認爲success。
failureThreshold：最少連續幾回探測失敗的次數，知足該次數則認爲failtcp

Handler:
不管是liveness仍是readiness都支持3種類型的探測方式：執行命令、http方式以及tcp方式。ide

// Handler defines a specific action that should be taken
// TODO: pass structured data to these actions, and document that data here.
type Handler struct {
    // One and only one of the following should be specified.
    // Exec specifies the action to take.
    // +optional
    Exec *ExecAction `json:"exec,omitempty" protobuf:"bytes,1,opt,name=exec"`
    // HTTPGet specifies the http request to perform.
    // +optional
    HTTPGet *HTTPGetAction `json:"httpGet,omitempty" protobuf:"bytes,2,opt,name=httpGet"`
    // TCPSocket specifies an action involving a TCP port.
    // TCP hooks not yet supported
    // TODO: implement a realistic TCP lifecycle hook
    // +optional
    TCPSocket *TCPSocketAction `json:"tcpSocket,omitempty" protobuf:"bytes,3,opt,name=tcpSocket"`
}

接下來看一下prober.go中的runProbe方法。

func (pb *prober) runProbe(probeType probeType, p *v1.Probe, pod *v1.Pod, status v1.PodStatus, container v1.Container, containerID kubecontainer.ContainerID) (probe.Result, string, error) {
    timeout := time.Duration(p.TimeoutSeconds) * time.Second
    if p.Exec != nil {
        glog.V(4).Infof("Exec-Probe Pod: %v, Container: %v, Command: %v", pod, container, p.Exec.Command)
        command := kubecontainer.ExpandContainerCommandOnlyStatic(p.Exec.Command, container.Env)
        return pb.exec.Probe(pb.newExecInContainer(container, containerID, command, timeout))
    }
    if p.HTTPGet != nil {
        scheme := strings.ToLower(string(p.HTTPGet.Scheme))
        host := p.HTTPGet.Host
        if host == "" {
            host = status.PodIP
        }
        port, err := extractPort(p.HTTPGet.Port, container)
        if err != nil {
            return probe.Unknown, "", err
        }
        path := p.HTTPGet.Path
        glog.V(4).Infof("HTTP-Probe Host: %v://%v, Port: %v, Path: %v", scheme, host, port, path)
        url := formatURL(scheme, host, port, path)
        headers := buildHeader(p.HTTPGet.HTTPHeaders)
        glog.V(4).Infof("HTTP-Probe Headers: %v", headers)
        if probeType == liveness {
            return pb.livenessHttp.Probe(url, headers, timeout)
        } else { // readiness
            return pb.readinessHttp.Probe(url, headers, timeout)
        }
    }
    if p.TCPSocket != nil {
        port, err := extractPort(p.TCPSocket.Port, container)
        if err != nil {
            return probe.Unknown, "", err
        }
        host := p.TCPSocket.Host
        if host == "" {
            host = status.PodIP
        }
        glog.V(4).Infof("TCP-Probe Host: %v, Port: %v, Timeout: %v", host, port, timeout)
        return pb.tcp.Probe(host, port, timeout)
    }
    glog.Warningf("Failed to find probe builder for container: %v", container)
    return probe.Unknown, "", fmt.Errorf("Missing probe handler for %s:%s", format.Pod(pod), container.Name)
}

1.執行命令方式
經過newExecInContainer方法調用CRI執行命令：oop

// ExecAction describes a "run in container" action.
type ExecAction struct {
    // Command is the command line to execute inside the container, the working directory for the
    // command  is root ('/') in the container's filesystem. The command is simply exec'd, it is
    // not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use
    // a shell, you need to explicitly call out to that shell.
    // Exit status of 0 is treated as live/healthy and non-zero is unhealthy.
    // +optional
    Command []string `json:"command,omitempty" protobuf:"bytes,1,rep,name=command"`
}

2.http GET方式
經過http GET方式進行探測。
Port：表示訪問容器的端口
Host：表示訪問的主機，默認是Pod IPui

// HTTPGetAction describes an action based on HTTP Get requests.
type HTTPGetAction struct {
    // Path to access on the HTTP server.
    // +optional
    Path string `json:"path,omitempty" protobuf:"bytes,1,opt,name=path"`
    // Name or number of the port to access on the container.
    // Number must be in the range 1 to 65535.
    // Name must be an IANA_SVC_NAME.
    Port intstr.IntOrString `json:"port" protobuf:"bytes,2,opt,name=port"`
    // Host name to connect to, defaults to the pod IP. You probably want to set
    // "Host" in httpHeaders instead.
    // +optional
    Host string `json:"host,omitempty" protobuf:"bytes,3,opt,name=host"`
    // Scheme to use for connecting to the host.
    // Defaults to HTTP.
    // +optional
    Scheme URIScheme `json:"scheme,omitempty" protobuf:"bytes,4,opt,name=scheme,casttype=URIScheme"`
    // Custom headers to set in the request. HTTP allows repeated headers.
    // +optional
    HTTPHeaders []HTTPHeader `json:"httpHeaders,omitempty" protobuf:"bytes,5,rep,name=httpHeaders"`
}

3.tcp方式
經過設置主機和端口便可進行tcp方式訪問url

// TCPSocketAction describes an action based on opening a socket
type TCPSocketAction struct {
    // Number or name of the port to access on the container.
    // Number must be in the range 1 to 65535.
    // Name must be an IANA_SVC_NAME.
    Port intstr.IntOrString `json:"port" protobuf:"bytes,1,opt,name=port"`
    // Optional: Host name to connect to, defaults to the pod IP.
    // +optional
    Host string `json:"host,omitempty" protobuf:"bytes,2,opt,name=host"`
}

此處腦洞一下：若是三種探測方式都設置了，會如何執行處理？

思考

經過k8s部署生產環境應用時，建議設置上liveness和readiness, 這也是保障服務穩定性的最佳實踐。
另外因爲Pod Ready不能保證明際的業務應用Ready可用，在最新的 1.14 版本中新增了一個Pod Readiness Gates 特性 。經過這個特性，能夠保證應用Ready後進而設置Pod Ready。

結尾

針對上面的腦洞：若是三種探測方式都設置了，會如何執行處理？
答：咱們若是在Pod中設置多個探測方式，提交配置的時候會直接報錯：
undefined
此處繼續源代碼：在validation.go中validateHandler中進行了限制(也爲上面Handler struct提到的"One and only one of the following should be specified."提供了事實依據)

func validateHandler(handler *core.Handler, fldPath *field.Path) field.ErrorList {
    numHandlers := 0
    allErrors := field.ErrorList{}
    if handler.Exec != nil {
        if numHandlers > 0 {
            allErrors = append(allErrors, field.Forbidden(fldPath.Child("exec"), "may not specify more than 1 handler type"))
        } else {
            numHandlers++
            allErrors = append(allErrors, validateExecAction(handler.Exec, fldPath.Child("exec"))...)
        }
    }
    if handler.HTTPGet != nil {
        if numHandlers > 0 {
            allErrors = append(allErrors, field.Forbidden(fldPath.Child("httpGet"), "may not specify more than 1 handler type"))
        } else {
            numHandlers++
            allErrors = append(allErrors, validateHTTPGetAction(handler.HTTPGet, fldPath.Child("httpGet"))...)
        }
    }
    if handler.TCPSocket != nil {
        if numHandlers > 0 {
            allErrors = append(allErrors, field.Forbidden(fldPath.Child("tcpSocket"), "may not specify more than 1 handler type"))
        } else {
            numHandlers++
            allErrors = append(allErrors, validateTCPSocketAction(handler.TCPSocket, fldPath.Child("tcpSocket"))...)
        }
    }
    if numHandlers == 0 {
        allErrors = append(allErrors, field.Required(fldPath, "must specify a handler type"))
    }
    return allErrors
}


原文連接 本文爲雲棲社區原創內容，未經容許不得轉載。

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程