深刻分析Kubelet的動態配置

時間  2019-11-13
標籤 深刻 分析 kubelet 動態 配置 简体版
原文   原文鏈接

Author: xidianwangtao@gmail.com Kubernetes 1.13node

摘要：Kubelet動態配置可使讓咱們及其方便的大規模更新集羣Kubelet配置，讓咱們能夠像配置集羣中其餘應用同樣經過ConfigMap配置Kubelet，而且Kubelet能動態感知到配置的變化，自動退出從新加載最新配置。不只如此，Kubelet Dynamic Config還有本地Checkpoint數據、失敗回滾到上一個可用配置集等美麗特性。本文介紹了Kubelet的配置組成部分、簡要工做流，以及核心機制(BootStrap、Sync)的實現原理、目前還有待完善的地方等。docker

Kubelet Configuration

Kubelet配置分兩部分:api

  • KubeletFlag: 指那些不容許在kubelet運行時進行修改的配置集，或者不能在集羣中各個Nodes之間共享的配置集。

  • KubeletConfiguration: 指能夠在集羣中各個Nodes之間共享的配置集。

Dynamic Kubelet Config

Core Features

  • Kubelet attempts to use the dynamically assigned configuration.安全

  • Kubelet 「checkpoints」 configuration to local disk, enabling restarts without API server access.app

  • Kubelet reports assigned, active, and last-known-good configuration sources in the Node status.異步

  • When invalid configuration is dynamically assigned, Kubelet automatically falls back to a last-known-good configuration and reports errors in the Node status.ide

Workflow

The basic workflow for configuring a Kubelet in a live cluster is as follows:oop

  • Each Kubelet watches its associated Node object for changes.測試

  • When the Node.Spec.ConfigSource.ConfigMap reference is updated, the Kubelet will 「checkpoint」 the new ConfigMap by writing the files it contains to local disk.this

  • The Kubelet will then exit, and the OS-level process manager will restart it.

Note that if the Node.Spec.ConfigSource.ConfigMap reference is not set, the Kubelet uses the set of flags and config files local to the machine it is running on.

  • Once restarted, the Kubelet will attempt to use the configuration from the new checkpoint.

  • If the new configuration passes the Kubelet’s internal validation, the Kubelet will update Node.Status.Config to reflect that it is using the new configuration.

  • If the new configuration is invalid, the Kubelet will fall back to its last-known-good configuration and report an error in Node.Status.Config.

  • Note that the default last-known-good configuration is the combination of Kubelet command-line flags with the Kubelet’s local configuration file. Command-line flags that overlap with the config file always take precedence over both the local configuration file and dynamic configurations, for backwards-compatibility.

The status of the Node’s Kubelet configuration is reported via Node.Status.Config. Once you have updated a Node to use the new ConfigMap, you can observe this status to confirm that the Node is using the intended configuration.

更多使用簡介請參考：https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

Enable and New Dynamic Kubelet Config Controller

在Kubelet構建Command時(NewKubeletCommand),會對kubelet的配置進行分析、加載。

func NewKubeletCommand(stopCh <-chan struct{}) *cobra.Command {
...
	// load kubelet config file, if provided
	if configFile := kubeletFlags.KubeletConfigFile; len(configFile) > 0 {
		kubeletConfig, err = loadConfigFile(configFile)
		...
		if err := kubeletConfigFlagPrecedence(kubeletConfig, args); err != nil {
			klog.Fatal(err)
		}
		// update feature gates based on new config
		if err := utilfeature.DefaultFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {
			klog.Fatal(err)
		}
	}
	
	// We always validate the local configuration (command line + config file).
	// This is the default "last-known-good" config for dynamic config, and must always remain valid.
	if err := kubeletconfigvalidation.ValidateKubeletConfiguration(kubeletConfig); err != nil {
		klog.Fatal(err)
	}
	
	// use dynamic kubelet config, if enabled
	var kubeletConfigController *dynamickubeletconfig.Controller
	if dynamicConfigDir := kubeletFlags.DynamicConfigDir.Value(); len(dynamicConfigDir) > 0 {
		var dynamicKubeletConfig *kubeletconfiginternal.KubeletConfiguration
		dynamicKubeletConfig, kubeletConfigController, err = BootstrapKubeletConfigController(dynamicConfigDir,
			func(kc *kubeletconfiginternal.KubeletConfiguration) error {
				return kubeletConfigFlagPrecedence(kc, args)
			})
		if err != nil {
			klog.Fatal(err)
		}
		// If we should just use our existing, local config, the controller will return a nil config
		if dynamicKubeletConfig != nil {
			kubeletConfig = dynamicKubeletConfig
			// Note: flag precedence was already enforced in the controller, prior to validation,
			// by our above transform function. Now we simply update feature gates from the new config.
			if err := utilfeature.DefaultFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {
				klog.Fatal(err)
			}
		}
	}
	// construct a KubeletServer from kubeletFlags and kubeletConfig
	kubeletServer := &options.KubeletServer{
		KubeletFlags:         *kubeletFlags,
		KubeletConfiguration: *kubeletConfig,
	}

	// use kubeletServer to construct the default KubeletDeps
	kubeletDeps, err := UnsecuredDependencies(kubeletServer)
	if err != nil {
		klog.Fatal(err)
	}

	// add the kubelet config controller to kubeletDeps
	kubeletDeps.KubeletConfigController = kubeletConfigController
	
	// start the experimental docker shim, if enabled
	if kubeletServer.KubeletFlags.ExperimentalDockershim {
		if err := RunDockershim(&kubeletServer.KubeletFlags, kubeletConfig, stopCh); err != nil {
			klog.Fatal(err)
		}
		return
	}

	// run the kubelet
	klog.V(5).Infof("KubeletConfiguration: %#v", kubeletServer.KubeletConfiguration)
	if err := Run(kubeletServer, kubeletDeps, stopCh); err != nil {
		klog.Fatal(err)
	}
...
}

  • 首先會檢查--config參數是否配置了，若是配置了，則讀取該文件內容並轉換成KubeletConfiguration，並調用kubeletConfigFlagPrecedence將其與GlobalFlags、Command Line Flags等進行整合。後面分析Dynamic Kubelet Config Controller BootStrap時會對kubeletConfigFlagPrecedence進行詳細說明。

  • 而後對上面整合獲得的KubeletConfig中各個參數進行合法有效性校驗。

  • 若是kubelet配置了--dynamic-config-dir而且Enable了DynamicKubeletConfig FeatureGates（默認Disable），則會建立Dynamic Kubelet Config Controller，並調用其BootStrap方法進行配置初始化。注意，這裏給Controller設置的TransformFunc爲kubeletConfigFlagPrecedence，後面會分析。

type Controller struct {
	transform TransformFunc

	// pendingConfigSource; write to this channel to indicate that the config source needs to be synced from the API server
	pendingConfigSource chan bool

	// configStatus manages the status we report on the Node object
	configStatus status.NodeConfigStatus

	// nodeInformer is the informer that watches the Node object
	nodeInformer cache.SharedInformer

	// remoteConfigSourceInformer is the informer that watches the assigned config source
	remoteConfigSourceInformer cache.SharedInformer

	// checkpointStore persists config source checkpoints to a storage layer
	checkpointStore store.Store
}

  • 若是Dynamic Kubelet Config Controller的BootStrap返回的DynamicKubeletConfig不爲空，那麼原來從--config中加載的KubeletConfig配置將被丟棄，使用DynamicKubeletConfig。

  • 將 Dynamic Kubelet Config Controller添加到Kubelet Dependencies中。Kubelet Dependencies是Kubelet運行所依賴的接口，包括cAdvisor接口、Dockershim ClientConfig接口、VolumePlugin接口、DynamicPluginProber接口、OOMAdjuster接口等共20個組件接口。

Dynamic Kubelet Config Controller BootStrap

NewKubeletCommand時，會建立Dynamic Kubelet Config Controller並調用其BootStrap方法完成Kubelet Dynamic Config的Checkpoints加載和Node ConfigStatus的更新。下面將詳細分析BootStrap的原理。

// Bootstrap attempts to return a valid KubeletConfiguration based on the configuration of the Controller,
// or returns an error if no valid configuration could be produced. Bootstrap should be called synchronously before StartSync.
// If the pre-existing local configuration should be used, Bootstrap returns a nil config.
func (cc *Controller) Bootstrap() (*kubeletconfig.KubeletConfiguration, error) {
	utillog.Infof("starting controller")

	// ensure the filesystem is initialized
	if err := cc.initializeDynamicConfigDir(); err != nil {
		return nil, err
	}

	// determine assigned source and set status
	assignedSource, err := cc.checkpointStore.Assigned()
	if err != nil {
		return nil, err
	}
	if assignedSource != nil {
		cc.configStatus.SetAssigned(assignedSource.NodeConfigSource())
	}

	// determine last-known-good source and set status
	lastKnownGoodSource, err := cc.checkpointStore.LastKnownGood()
	if err != nil {
		return nil, err
	}
	if lastKnownGoodSource != nil {
		cc.configStatus.SetLastKnownGood(lastKnownGoodSource.NodeConfigSource())
	}

	// if the assigned source is nil, return nil to indicate local config
	if assignedSource == nil {
		return nil, nil
	}

	// attempt to load assigned config
	assignedConfig, reason, err := cc.loadConfig(assignedSource)
	if err == nil {
		// update the active source to the non-nil assigned source
		cc.configStatus.SetActive(assignedSource.NodeConfigSource())

		go wait.Forever(func() { cc.checkTrial(configTrialDuration) }, 10*time.Second)

		return assignedConfig, nil
	} // Assert: the assigned config failed to load or validate

	utillog.Errorf(fmt.Sprintf("%s, error: %v", reason, err))

	// set status to indicate the failure with the assigned config
	cc.configStatus.SetError(reason)

	// if the last-known-good source is nil, return nil to indicate local config
	if lastKnownGoodSource == nil {
		return nil, nil
	}

	// attempt to load the last-known-good config
	lastKnownGoodConfig, _, err := cc.loadConfig(lastKnownGoodSource)
	if err != nil {
		// we failed to load the last-known-good, so something is really messed up and we just return the error
		return nil, err
	}

	// set status to indicate the active source is the non-nil last-known-good source
	cc.configStatus.SetActive(lastKnownGoodSource.NodeConfigSource())
	return lastKnownGoodConfig, nil
}

BootStrap中關鍵的幾個流程是：

  • Initialize Dynamic Config Dir

  • Get Assigned NodeConfigSource From Checkpoints

  • Get LastKnowsGood(LKG) NodeConfigSource From Checkpoints

  • Load Assigned Config

  • Load LastKnownGood Config If Necessary

Initialize Dynamic Config Dir

確保如下目錄已經建立，若是沒有，則自動建立：

  • {dynamic-config-dir}/store/: Dynamic Config Dir的根目錄；

  • {dynamic-config-dir}/store/meta/: Dynamic Config Dir的meta目錄，包含assigned和last-known-good兩個子目錄；

  • {dynamic-config-dir}/store/meta/assigned: Kubelet將要試圖加載並使用的Checkpointed NodeConfigSource，文件內容是kubeletconfigv1beta1.SerializedNodeConfigSource的Yaml格式；

  • {dynamic-config-dir}/store/meta/last-known-good: Kubelet在使用Assigned NodeConfigSource異常時，但願回滾的NodeConfigSource，文件內容是kubeletconfigv1beta1.SerializedNodeConfigSource的Yaml格式。

  • {dynamic-config-dir}/store/checkpoints/: 記錄各個Checkpoints的ConfigMap中的Kubelet配置內容，它下面還有兩個子目錄，分別是{ReferencedConfigMap-UID}和{ReferencedConfigMap-ResourceVersion}，最外層是文件名爲{node.spec.configSource.configMap.data}的key（一般設置爲kubelet），文件中內容是data.key對應的value，也就是純粹的kubelet配置項。

Get Assigned NodeConfigSource From Checkpoints

  • 從Node{dynamic-config-dir}/store/meta/assigned文件中讀取Assigned NodeConfigSource，並設置到nodeConfigStatus.Assigned中。

  • 若是讀取到的讀取Assigned NodeConfigSource爲空，則BootStrape返回nil，表示須要load config。

Get LastKnowsGood(LKG) NodeConfigSource From Checkpoints

  • 從Node{dynamic-config-dir}/store/meta/last-known-good/目錄中讀取Last-Known-Good NodeConfigSource，並設置到nodeConfigStatus.LastKnownGood中。

Load Assigned Config

  • 若是讀取到的讀取Assigned NodeConfigSource爲非空，則從Checkpoints中加載對應的Config，並確保能成功轉換成KubeletConfiguration類型。

    • 檢查Assigned NodeConfigSource引用的ConfigMap的UID和ResourceVersion是否爲空；

    • 檢查目錄{dynamic-config-dir}/store/checkpoints/{ReferencedConfigMap-UID}/{ReferencedConfigMap-ResourceVersion}/是否存在；

    • 當以上條件都知足時，加載{dynamic-config-dir}/store/checkpoints/{ReferencedConfigMap-UID}/{ReferencedConfigMap-ResourceVersion}/{ReferencedConfigMap-KubeletKey}並進行kubeletconfig.KubeletConfiguration格式轉換後返回。

      • {ReferencedConfigMap-KubeletKey}一般咱們寫成kubelet。

  • 把Load出來的KubeletConfiguration，經過transform（kubeletConfigFlagPrecedence）與其餘Flags進行整合，最終造成kubelet啓動的完整配置。整合邏輯以下：

    • 附加 Global Flags(logtostderr, v, log_dir, log_file等，)

    • 附加 Credential Provider Flags（azure-container-registry-config）

    • 附加 Version Flag(version)

    • 附加 log Flush Frequency Flag(log-flush-frequency)

    • 整合KubeletConfiguration和Kubelet Config Flags (cmd/kubelet/app/options/options.go:459中枚舉了全部Kubelet Config Flags)，注意這裏KubeletConfiguration的FeatureGates與DefaultFeatureGate進行了合併。

    • 再附加Command Line中的kubelet參數。

    • 最後把KubeletConfiguration的FeatureGates加回到Kubelet Config Flags中，也就是說，若是KubeletConfiguration的FeatureGates與DefaultFeatureGate有衝突時，之前者爲準。

  • 對transform後的KubeletConfiguration中的相關配置進行合法性校驗，好比：

    • --enforce-node-allocatable中的key只能是pods,system-resverved,kube-reserved,none類型之一。

    • --hairpin-mode的值只能是hairpin-veth, promiscuous-bridge,none之一。

  • 若是以上Load Config過程當中一切正常，接下來就會：

    • 將前面合法有效的Assigned NodeConfigSource更新到Node ConfigStatus的Active字段中，表示啓用該Assigned NodeConfigSource。

    • 而後異步啓動協程：每10s調用checkTrial檢查Assigned Path（即{dynamic-config-dir}/store/meta/assigned）的最近修改時間是否已通過了10min（hard code，暫不可配置），若是不到10min，則checkTrial不作任何處理，繼續等待下一個10s循環觸發checkTrial，若是已經超過10min，處理以下。CheckTrial的目的是給Node預留一些容忍重啓次數，或者Master協同時間。

      • 檢查Assigned NodeConfigSource和當前checkpoints中記錄LKG NodeConfigSource（{dynamic-config-dir}/store/meta/last-known-good/）是否內容一致；

      • 若是不一致，則將Assigned NodeConfigSource更新到checkpoints中的LKG NodeConfigSource（{dynamic-config-dir}/store/meta/last-known-good/），並將Assigned NodeConfigSource更新Node ConfigStatus的LastKnownGood字段。

      • 若是一致，則checkTrial不須要作任何處理。

    • BootStrap結束，返回該Assigned NodeConfigSource。

Load LastKnownGood Config If Necessary

當Load Assigned NodeConfigSource過程有異常，說明Assigned配置有問題，則須要回滾到LastKnownGood Config。

  • 若是前面的Load Assigned NodeConfigSource過程有異常，將異常的reason信息更新到Node ConfigStatus.Error中。而後開始嘗試Load前面讀取到的Last-Known-Good NodeConfigSource。若是前面讀取到的讀取Last-Known-Good NodeConfigSource爲空，則BootStrape返回nil，表示須要load local config。

  • Load LKG NodeConfigSource的邏輯與Load Assigned NodeConfigSource徹底同樣，都是調用Controller.loadConfig(source checkpoint.RemoteConfigSource) (*kubeletconfig.KubeletConfiguration, string, error)進行的。

  • 若是Load LKG NodeConfigSource成功，將LKG NodeConfigSource更新到Node ConfigStatus.Active字段中，BootStrap返回LKG NodeConfigSource。

  • 若是Load LKG NodeConfigSource失敗，則返回nil和error。

StartSync

BootStrap只是完成Kubelet啓動時Checkpointed Dynamic Config的處理和加載。做爲一個Controller，它還須要不斷的與API Server進行Sync，當Referenced ConfigMap發生變動時，完成Download並Checkpoints等一系列控制。

在Kubelet run時，檢查如下條件都知足的狀況下，會調用KubeletConfigController StartSync開始其主控制邏輯:

  • DynamicKubeletConfig FeatureGates是否Enable；

  • 配置了--dynamic-config-dir；

  • KubeletConfigController成功添加到了Kubelet Dependencies中；

  • Kubelet以非standalone模式運行（即有效配置了KubeConfig）；

  • Kubelet以非Runonce模式運行（即--runonce爲false或者沒有配置）；

// StartSync tells the controller to start the goroutines that sync status/config to/from the API server.
// The clients must be non-nil, and the nodeName must be non-empty.
func (cc *Controller) StartSync(client clientset.Interface, eventClient v1core.EventsGetter, nodeName string) error {
	const errFmt = "cannot start Kubelet config sync: %s"
	if client == nil {
		return fmt.Errorf(errFmt, "nil client")
	}
	if eventClient == nil {
		return fmt.Errorf(errFmt, "nil event client")
	}
	if nodeName == "" {
		return fmt.Errorf(errFmt, "empty nodeName")
	}

	// status sync worker
	statusSyncLoopFunc := utilpanic.HandlePanic(func() {
		utillog.Infof("starting status sync loop")
		wait.JitterUntil(func() {
			cc.configStatus.Sync(client, nodeName)
		}, 10*time.Second, 0.2, true, wait.NeverStop)
	})
	// remote config source informer, if we have a remote source to watch
	assignedSource, err := cc.checkpointStore.Assigned()
	if err != nil {
		return fmt.Errorf(errFmt, err)
	} else if assignedSource == nil {
		utillog.Infof("local source is assigned, will not start remote config source informer")
	} else {
		cc.remoteConfigSourceInformer = assignedSource.Informer(client, cache.ResourceEventHandlerFuncs{
			AddFunc:    cc.onAddRemoteConfigSourceEvent,
			UpdateFunc: cc.onUpdateRemoteConfigSourceEvent,
			DeleteFunc: cc.onDeleteRemoteConfigSourceEvent,
		},
		)
	}
	remoteConfigSourceInformerFunc := utilpanic.HandlePanic(func() {
		if cc.remoteConfigSourceInformer != nil {
			utillog.Infof("starting remote config source informer")
			cc.remoteConfigSourceInformer.Run(wait.NeverStop)
		}
	})
	// node informer
	cc.nodeInformer = newSharedNodeInformer(client, nodeName,
		cc.onAddNodeEvent, cc.onUpdateNodeEvent, cc.onDeleteNodeEvent)
	nodeInformerFunc := utilpanic.HandlePanic(func() {
		utillog.Infof("starting Node informer")
		cc.nodeInformer.Run(wait.NeverStop)
	})
	// config sync worker
	configSyncLoopFunc := utilpanic.HandlePanic(func() {
		utillog.Infof("starting Kubelet config sync loop")
		wait.JitterUntil(func() {
			cc.syncConfigSource(client, eventClient, nodeName)
		}, 10*time.Second, 0.2, true, wait.NeverStop)
	})

	go statusSyncLoopFunc()
	go remoteConfigSourceInformerFunc()
	go nodeInformerFunc()
	go configSyncLoopFunc()
	return nil
}

Start Node ConfigStatus Sync Worker

KubeletConfigController建立的時會invoke NewNodeConfigStatus，那時就會往nodeConfigStatus.syncCh中傳入true，開始ConfigStatus的Sync操做。

// Sync attempts to sync the status with the Node object for this Kubelet,
// if syncing fails, an error is logged, and work is queued for retry.
func (s *nodeConfigStatus) Sync(client clientset.Interface, nodeName string) {
	select {
	case <-s.syncCh:
	default:
		// no work to be done, return
		return
	}

	utillog.Infof("updating Node.Status.Config")

	// grab the lock
	s.mux.Lock()
	defer s.mux.Unlock()

	// if the sync fails, we want to retry
	var err error
	defer func() {
		if err != nil {
			utillog.Errorf(err.Error())
			s.sync()
		}
	}()

	// get the Node so we can check the current status
	oldNode, err := client.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{})
	if err != nil {
		err = fmt.Errorf("could not get Node %q, will not sync status, error: %v", nodeName, err)
		return
	}

	status := &s.status
	// override error, if necessary
	if len(s.errorOverride) > 0 {
		// copy the status, so we don't overwrite the prior error
		// with the override
		status = status.DeepCopy()
		status.Error = s.errorOverride
	}

	...

	// apply the status to a copy of the node so we don't modify the object in the informer's store
	newNode := oldNode.DeepCopy()
	newNode.Status.Config = status

	// patch the node with the new status
	if _, _, err := nodeutil.PatchNodeStatus(client.CoreV1(), types.NodeName(nodeName), oldNode, newNode); err != nil {
		utillog.Errorf("failed to patch node status, error: %v", err)
	}
}

  • ConfigStatus Sync Worker啓動後每隔必定時間（[10s, 12s]中的一個隨機數）啓動一次，循環如此。

  • worker從API Server獲取本節點對應的Node，將ConfigStatus調用Patch接口更新到Node.Status.Config中。

  • 若是Sync失敗，則會觸發從新Sync。

Start Assigned NodeConfigSource Informer

  • Assigned NodeConfigSource Informer經過FieldSelector List指定name的ConfigMap對象，resyncPeriod爲[15m,30m]中的一個隨機值。

  • 註冊Add/Update/Delete Event Handler分別爲onAddRemoteConfigSourceEvent/onUpdateRemoteConfigSourceEvent/onDeleteRemoteConfigSourceEvent;

    • Add/Update/Delete時，則invoke pokeConfigSourceWorker往KubeletConfigController.pendingConfigSource channel傳入true，觸發Config Sync Worker從API Server同步最新的ConfigMap download and save到Checkpoints, 並中止kubelet待systemd重啓。

    • Delete時，稍有不一樣的是，check in-memory 或者 download from api都會失敗，此時只更新ConfigStatus.Error，並不會有後續的save to checkpoints、更新meta/assigned及中止kubelet的操做。

Start Node Informer

  • Node Informer經過FieldSelector List指定name的該nodename對象，resyncPeriod爲[15m,30m]中的一個隨機值。

  • 註冊Add/Update/Delete Event Handler分別爲onAddNodeEvent/onUpdateNodeEvent/onDeleteNodeEvent;

    • Add/Update時，則invoke pokeConfigSourceWorker往KubeletConfigController.pendingConfigSource channel傳入true，觸發Config Sync Worker從API Server同步最新的ConfigMap download and save到Checkpoints, 並中止kubelet待systemd重啓。

    • Delete時，並不會觸發任何實際的操做，好比清空{dynamic-config-dir}/store/下面的任何子目錄，包括checkpionts。因此，當你kubectl delete node $nodeName而後再kubectl add node $nodeName時，由onAddNodeEvent觸發pokeConfigSourceWorker進行Referenced ConfigMap的Sync工做。

Start Config Sync Worker

Config Sync Worker是Dynamic Kubelet Config Controller最爲核心的的Worker，它負責檢查檢查配置是否有更新，若是有更新還須要進行最新配置的下載及checkpointstore的更新、kubelet進程的退出等。

// syncConfigSource checks if work needs to be done to use a new configuration, and does that work if necessary
func (cc *Controller) syncConfigSource(client clientset.Interface, eventClient v1core.EventsGetter, nodeName string) {
	select {
	case <-cc.pendingConfigSource:
	default:
		// no work to be done, return
		return
	}

	// if the sync fails, we want to retry
	var syncerr error
	defer func() {
		if syncerr != nil {
			utillog.Errorf(syncerr.Error())
			cc.pokeConfigSourceWorker()
		}
	}()

	// get the latest Node.Spec.ConfigSource from the informer
	source, err := latestNodeConfigSource(cc.nodeInformer.GetStore(), nodeName)
	if err != nil {
		cc.configStatus.SetErrorOverride(fmt.Sprintf(status.SyncErrorFmt, status.InternalError))
		syncerr = fmt.Errorf("%s, error: %v", status.InternalError, err)
		return
	}

	// a nil source simply means we reset to local defaults
	if source == nil {
		utillog.Infof("Node.Spec.ConfigSource is empty, will reset assigned and last-known-good to defaults")
		if updated, reason, err := cc.resetConfig(); err != nil {
			reason = fmt.Sprintf(status.SyncErrorFmt, reason)
			cc.configStatus.SetErrorOverride(reason)
			syncerr = fmt.Errorf("%s, error: %v", reason, err)
			return
		} else if updated {
			restartForNewConfig(eventClient, nodeName, nil)
		}
		return
	}

	// construct the interface that can dynamically dispatch the correct Download, etc. methods for the given source type
	remote, reason, err := checkpoint.NewRemoteConfigSource(source)
	if err != nil {
		reason = fmt.Sprintf(status.SyncErrorFmt, reason)
		cc.configStatus.SetErrorOverride(reason)
		syncerr = fmt.Errorf("%s, error: %v", reason, err)
		return
	}

	// "download" source, either from informer's in-memory store or directly from the API server, if the informer doesn't have a copy
	payload, reason, err := cc.downloadConfigPayload(client, remote)
	if err != nil {
		reason = fmt.Sprintf(status.SyncErrorFmt, reason)
		cc.configStatus.SetErrorOverride(reason)
		syncerr = fmt.Errorf("%s, error: %v", reason, err)
		return
	}

	// save a checkpoint for the payload, if one does not already exist
	if reason, err := cc.saveConfigCheckpoint(remote, payload); err != nil {
		reason = fmt.Sprintf(status.SyncErrorFmt, reason)
		cc.configStatus.SetErrorOverride(reason)
		syncerr = fmt.Errorf("%s, error: %v", reason, err)
		return
	}

	// update the local, persistent record of assigned config
	if updated, reason, err := cc.setAssignedConfig(remote); err != nil {
		reason = fmt.Sprintf(status.SyncErrorFmt, reason)
		cc.configStatus.SetErrorOverride(reason)
		syncerr = fmt.Errorf("%s, error: %v", reason, err)
		return
	} else if updated {
		restartForNewConfig(eventClient, nodeName, remote)
	}

	cc.configStatus.SetErrorOverride("")
}

  • ConfigStatus Sync Worker啓動後每隔必定時間（[10s, 12s]中的一個隨機數）啓動一次，循環如此。

  • 從NodeInformer中獲取Node最新的NodeConfigSource(.Spec.ConfigSource.)，若是發生異常，把error更新到ConfigStatus.Error中並return，並不會觸發配置的回滾。

  • 若是獲取到Node最新的NodeConfigSource爲空，則invoke resetConfig將本地checkpointstore的assigned和last-known-good設置爲空，若是在次以前assigned原本不爲空，則invoke restartForNewConfig將kubelet進程退出並上報Event。

    • makeEvent(nodeName, apiv1.EventTypeNormal, KubeletConfigChangedEventReason, message)，其中message爲"Kubelet restarting to use local config"。

    • kubelet由systemd從新啓動時，會KubeletConfigController.BootStrap返回的KubeletConfiguration爲空，所以KubeletConfig會由--config主導。

  • 若是獲取到Node最新的NodeConfigSource不爲空，則invoke downloadConfigPayload從本地in-memory store或者API Server中download對應的ConfigMap並封裝到checkpoint.configMapPayload對象，而且更新該NodeConfigSource對象的UID和ResourceVersion。

  • 而後invoke saveConfigCheckpoint：

    • 先檢查該NodeConfigSource對應的{dynamic-config-dir}/store/checkpoints/{ReferencedConfigMap-UID}/{ReferencedConfigMap-ResourceVersion}/目錄是否存在。若是已經存在，說明checkpoint已經有了，則return，結束saveConfigCheckpoint操做。

    • 不然，invoke checkpiontStore.Save建立對應對應的{dynamic-config-dir}/store/checkpoints/{ReferencedConfigMap-UID}/{ReferencedConfigMap-ResourceVersion}/目錄，而且將ConfigMap.data(map[string]string)的Value字符串保存到文件名爲data.key的文件中，咱們一般以kubelet爲Key，也就是保存到{dynamic-config-dir}/store/checkpoints/{ReferencedConfigMap-UID}/{ReferencedConfigMap-ResourceVersion}/kubelet文件中。

  • saveConfigCheckpoint成功後，invoke setAssignedConfig更新checkpointStore Assigned({dynamic-config-dir}/store/meta/assigned文件，若是assigned先後內容確實發生了變化，則退出kubelet進程。

  • 若是以上步驟中，發生error，表示Sync失敗，則會觸發從新Sync。

當前不足

Kubelet Dynamic Config在Kubernetes 1.13中仍然仍是Alpha，其中有幾個關鍵點還沒解決：

  • 沒有提供原生的集羣灰度能力，須要用戶本身實現自動化灰度節點配置。若是全部Node引用同一個Kubelet ConfigMap，當該ConfigMap發生錯誤變動後，可能會致使集羣短期不可用。

    • 分批灰度全部Nodes的能力

    • 或者是滾動灰度全部Nodes的能力

  • 哪些集羣配置能夠經過Kubelet Dynamic Config安全可靠的動態變動，尚未一個徹底明確的集合。一般狀況下，咱們能夠參考staging/src/k8s.io/kubelet/config/v1beta1/types.go:62中對KubeletConfiguration的定義註解瞭解那些Dynamic Config，可是仍是建議在測試集羣中測試事後，再經過Dynamic Config灰度到生產環境。

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程