深刻分析Kubernetes Critical Pod（三）

本文介紹了Kubelet在Predicate Admit准入檢查時對CriticalPod的資源搶佔的原理，以及Priority Admission Controller對CriticalPod的PriorityClassName特殊處理。

深刻分析Kubernetes Critical Pod系列： 深刻分析Kubernetes Critical Pod（一） 深刻分析Kubernetes Critical Pod（二） 深刻分析Kubernetes Critical Pod（三） 深刻分析Kubernetes Critical Pod（四）

Kubelet Predicate Admit時對Critical的資源搶佔處理

kubelet 在Predicate Admit流程中，會對Pods進行各類Predicate准入檢查，包括GeneralPredicates檢查本節點是否有足夠的cpu,mem,gpu資源。若是GeneralPredicates准入檢測失敗，對於nonCriticalPod則直接Admit失敗，但若是是CriticalPod則會觸發kubelet preemption進行資源搶佔，按照必定規則殺死一些Pods釋放資源，搶佔成功，則Admit成功。

流程的源頭應該從kubelet初始化的流程開始。

pkg/kubelet/kubelet.go:315

// NewMainKubelet instantiates a new Kubelet object along with all the required internal modules.
// No initialization of Kubelet and its modules should happen here.
func NewMainKubelet(...) (*Kubelet, error) {
	...
   criticalPodAdmissionHandler := preemption.NewCriticalPodAdmissionHandler(klet.GetActivePods, killPodNow(klet.podWorkers, kubeDeps.Recorder), kubeDeps.Recorder)
	klet.admitHandlers.AddPodAdmitHandler(lifecycle.NewPredicateAdmitHandler(klet.getNodeAnyWay, criticalPodAdmissionHandler, klet.containerManager.UpdatePluginResources))
	// apply functional Option's
	for _, opt := range kubeDeps.Options {
		opt(klet)
	}

	...
	return klet, nil
}

在NewMainKubelet對kubelet進行初始化時，經過AddPodAdmitHandler註冊了criticalPodAdmissionHandler，CriticalPod的Admit的特殊之處就體如今criticalPodAdmissionHandler。

而後，咱們進入kubelet的predicateAdmitHandler流程中，看看GeneralPredicates失敗後的處理邏輯。

pkg/kubelet/lifecycle/predicate.go:58

func (w *predicateAdmitHandler) Admit(attrs *PodAdmitAttributes) PodAdmitResult {
	...

	fit, reasons, err := predicates.GeneralPredicates(podWithoutMissingExtendedResources, nil, nodeInfo)
	if err != nil {
		message := fmt.Sprintf("GeneralPredicates failed due to %v, which is unexpected.", err)
		glog.Warningf("Failed to admit pod %v - %s", format.Pod(pod), message)
		return PodAdmitResult{
			Admit:   fit,
			Reason:  "UnexpectedAdmissionError",
			Message: message,
		}
	}
	if !fit {
		fit, reasons, err = w.admissionFailureHandler.HandleAdmissionFailure(pod, reasons)
		if err != nil {
			message := fmt.Sprintf("Unexpected error while attempting to recover from admission failure: %v", err)
			glog.Warningf("Failed to admit pod %v - %s", format.Pod(pod), message)
			return PodAdmitResult{
				Admit:   fit,
				Reason:  "UnexpectedAdmissionError",
				Message: message,
			}
		}
	}
	...
	return PodAdmitResult{
		Admit: true,
	}
}

在kubelet predicateAdmitHandler中對Pod進行GeneralPredicates檢查cpu,mem,gpu資源時，若是發現資源不足致使Admit失敗，則接着調用HandleAdmissionFailure進行額外處理。前提提到，kubelet初始化時註冊了criticalPodAdmissionHandler爲HandleAdmissionFailure。

CriticalPodAdmissionHandler struct定義以下：

pkg/kubelet/preemption/preemption.go:41

type CriticalPodAdmissionHandler struct {
	getPodsFunc eviction.ActivePodsFunc
	killPodFunc eviction.KillPodFunc
	recorder    record.EventRecorder
}

CriticalPodAdmissionHandler的HandleAdmissionFailure方法就是處理CriticalPod特殊的邏輯所在。

pkg/kubelet/preemption/preemption.go:66

// HandleAdmissionFailure gracefully handles admission rejection, and, in some cases,
// to allow admission of the pod despite its previous failure.
func (c *CriticalPodAdmissionHandler) HandleAdmissionFailure(pod *v1.Pod, failureReasons []algorithm.PredicateFailureReason) (bool, []algorithm.PredicateFailureReason, error) {
	if !kubetypes.IsCriticalPod(pod) || !utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) {
		return false, failureReasons, nil
	}
	// InsufficientResourceError is not a reason to reject a critical pod.
	// Instead of rejecting, we free up resources to admit it, if no other reasons for rejection exist.
	nonResourceReasons := []algorithm.PredicateFailureReason{}
	resourceReasons := []*admissionRequirement{}
	for _, reason := range failureReasons {
		if r, ok := reason.(*predicates.InsufficientResourceError); ok {
			resourceReasons = append(resourceReasons, &admissionRequirement{
				resourceName: r.ResourceName,
				quantity:     r.GetInsufficientAmount(),
			})
		} else {
			nonResourceReasons = append(nonResourceReasons, reason)
		}
	}
	if len(nonResourceReasons) > 0 {
		// Return only reasons that are not resource related, since critical pods cannot fail admission for resource reasons.
		return false, nonResourceReasons, nil
	}
	err := c.evictPodsToFreeRequests(admissionRequirementList(resourceReasons))
	// if no error is returned, preemption succeeded and the pod is safe to admit.
	return err == nil, nil, err
}

• 若是Pod不是CriticalPod，或者ExperimentalCriticalPodAnnotation Feature Gate是關閉的，則直接返回false，表示Admit失敗。
• 判斷Admit的failureReasons是否包含predicate.InsufficientResourceError，若是包含，則調用evictPodsToFreeRequests觸發kubelet preemption。注意這裏的搶佔不一樣於scheduler preemtion，不要混淆了。

evictPodsToFreeRequests就是kubelet preemption進行資源搶佔的邏輯實現，其核心就是調用getPodsToPreempt挑選合適的待殺死的Pods(podsToPreempt)。

pkg/kubelet/preemption/preemption.go:121

// getPodsToPreempt returns a list of pods that could be preempted to free requests >= requirements
func getPodsToPreempt(pods []*v1.Pod, requirements admissionRequirementList) ([]*v1.Pod, error) {
	bestEffortPods, burstablePods, guaranteedPods := sortPodsByQOS(pods)

	// make sure that pods exist to reclaim the requirements
	unableToMeetRequirements := requirements.subtract(append(append(bestEffortPods, burstablePods...), guaranteedPods...)...)
	if len(unableToMeetRequirements) > 0 {
		return nil, fmt.Errorf("no set of running pods found to reclaim resources: %v", unableToMeetRequirements.toString())
	}
	// find the guaranteed pods we would need to evict if we already evicted ALL burstable and besteffort pods.
	guarateedToEvict, err := getPodsToPreemptByDistance(guaranteedPods, requirements.subtract(append(bestEffortPods, burstablePods...)...))
	if err != nil {
		return nil, err
	}
	// Find the burstable pods we would need to evict if we already evicted ALL besteffort pods, and the required guaranteed pods.
	burstableToEvict, err := getPodsToPreemptByDistance(burstablePods, requirements.subtract(append(bestEffortPods, guarateedToEvict...)...))
	if err != nil {
		return nil, err
	}
	// Find the besteffort pods we would need to evict if we already evicted the required guaranteed and burstable pods.
	bestEffortToEvict, err := getPodsToPreemptByDistance(bestEffortPods, requirements.subtract(append(burstableToEvict, guarateedToEvict...)...))
	if err != nil {
		return nil, err
	}
	return append(append(bestEffortToEvict, burstableToEvict...), guarateedToEvict...), nil
}

kubelet preemtion時候挑選待殺死Pods的邏輯以下：

• 若是該Pod的某個Resource request quantity超過了如今的全部的bestEffortPods, burstablePods, guaranteedPods的該Resource request quantity，則podsToPreempt爲nil，意味着無合適Pods以釋放。
• 若是釋放全部bestEffortPods, burstablePods的資源都不足夠，則再挑選guaranteedPods（guarateedToEvict）。挑選的規則是：
  • 規則一：越少的Pods被釋放越好；
  • 規則二：釋放的資源越少越好；
  • 規則一的優先級比規則二高；
• 若是釋放全部bestEffortPods及guarateedToEvict的資源都不足夠，則再挑選burstablePods(burstableToEvict)。挑選的規則同上。
• 若是釋放全部burstableToEvict及guarateedToEvict的資源都不足夠，則再挑選bestEffortPods(bestEffortToEvict)。挑選的規則同上。

也就是說：Pod Resource QoS優先級越低的越先被搶佔，同一個QoS Level內挑選Pods按照以下規則：

• 規則一：越少的Pods被釋放越好；
• 規則二：釋放的資源越少越好；
• 規則一的優先級比規則二高；

Priority Admission Controller對CriticalPod的特殊處理

咱們先看看幾類特殊的、系統預留的CriticalPod：

• ClusterCriticalPod: PriorityClass Name是system-cluster-critical的Pod。
• NodeCriticalPod:PriorityClass Name是system-node-critical的Pod。

若是AdmissionController中啓動了Priority Admission Controller，那麼在建立Pod時對Priority的檢查也存在CriticalPod的特殊處理。

Priority Admission Controller主要做用是根據Pod中指定的PriorityClassName替換成對應的Spec.Pritory數值。

plugin/pkg/admission/priority/admission.go:138

// admitPod makes sure a new pod does not set spec.Priority field. It also makes sure that the PriorityClassName exists if it is provided and resolves the pod priority from the PriorityClassName.
func (p *priorityPlugin) admitPod(a admission.Attributes) error {
	operation := a.GetOperation()
	pod, ok := a.GetObject().(*api.Pod)
	if !ok {
		return errors.NewBadRequest("resource was marked with kind Pod but was unable to be converted")
	}

	// Make sure that the client has not set `priority` at the time of pod creation.
	if operation == admission.Create && pod.Spec.Priority != nil {
		return admission.NewForbidden(a, fmt.Errorf("the integer value of priority must not be provided in pod spec. Priority admission controller populates the value from the given PriorityClass name"))
	}
	if utilfeature.DefaultFeatureGate.Enabled(features.PodPriority) {
		var priority int32
		// TODO: @ravig - This is for backwards compatibility to ensure that critical pods with annotations just work fine.
		// Remove when no longer needed.
		if len(pod.Spec.PriorityClassName) == 0 &&
			utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) &&
			kubelettypes.IsCritical(a.GetNamespace(), pod.Annotations) {
			pod.Spec.PriorityClassName = scheduling.SystemClusterCritical
		}
		if len(pod.Spec.PriorityClassName) == 0 {
			var err error
			priority, err = p.getDefaultPriority()
			if err != nil {
				return fmt.Errorf("failed to get default priority class: %v", err)
			}
		} else {
			// Try resolving the priority class name.
			pc, err := p.lister.Get(pod.Spec.PriorityClassName)
			if err != nil {
				if errors.IsNotFound(err) {
					return admission.NewForbidden(a, fmt.Errorf("no PriorityClass with name %v was found", pod.Spec.PriorityClassName))
				}

				return fmt.Errorf("failed to get PriorityClass with name %s: %v", pod.Spec.PriorityClassName, err)
			}

			priority = pc.Value
		}
		pod.Spec.Priority = &priority
	}
	return nil
}

同時知足如下全部條件時，給Pod的Spec.PriorityClassName賦值爲system-cluster-critical,即認爲是ClusterCriticalPod。

• 若是Enable了ExperimentalCriticalPodAnnotation和PodPriority Feature Gate；
• 該Pod沒有指定PriorityClassName；
• 該Pod屬於kube-system namespace；
• 該Pod打了scheduler.alpha.kubernetes.io/critical-pod="" Annotation；

總結

本文介紹了Kubelet在Predicate Admit准入檢查時對CriticalPod的資源搶佔的原理，以及Priority Admission Controller對CriticalPod的PriorityClassName特殊處理。下一篇是最後一處關於Kubernetes對CriticalPod進行特殊待遇的地方——DaemonSet Controller。