istio1.7.3版本啓用ISTIO-CNI後istio-validation沒法啓動

啓用ISTIO-CNI後自動注入的POD會啓動istio-validation容器用來檢測網絡是否正常，在爲咱們公司另一條業務線的測試環境Setup時發現istio-validation容器沒法啓動，日誌輸出：

Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused

各類排查，最後查看系統日誌journalctl -ex

Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: W1102 14:50:30.291177    1029 cni.go:202] Error validating CNI config list {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "name": "cbr0",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "cniVersion": "0.3.1",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "plugins": [
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "type": "flannel",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "delegate": {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "hairpinMode": true,
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "isDefaultGateway": true
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: },
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "type": "portmap",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "capabilities": {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "portMappings": true
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: },
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "cniVersion": "0.3.1",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "name": "istio-cni",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "type": "istio-cni",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "log_level": "info",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "kubernetes": {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "kubeconfig": "/etc/cni/net.d/ZZZ-istio-cni-kubeconfig",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "cni_bin_dir": "/opt/cni/bin",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "exclude_namespaces": [
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "istio-system",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "kube-system"
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: ]
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: ]
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: : [failed to find plugin "istio-cni" in path [/opt/kube/bin]]
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: W1102 14:50:30.291194    1029 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d

發現是由於istio-cni的配置和K8S配置的cni可執行文件路徑不一致致使，istio-cni的demonset啓動的pod沒法調用該文件夾下的二進制文件建立IPTABLES規則，這種狀況比較容易出如今藉助各類第三方工具進行K8S集羣部署的環境中好比ansible部署k8s集羣，默認CNI可執行文件目錄在/opt/kube/bin而istio默認設置爲/opt/cni/bin,查看configmap或者istio-cni的pod日誌均可以找到

解決方案：

方案一：

修改部署istio都yaml文件加入官方說明的cniBinDir: 你的路徑

cni:
      excludeNamespaces:
       - istio-system
       - kube-system
      logLevel: info
      cniBinDir: /opt/kube/bin
      repair:
        enabled: true
        deletePods: false

或者命令行方式部署時加入--set values.cni.cniBinDir=... 和 --set values.cni.cniConfDir=... 選項

方案二：

修改istio-system空間下名爲istio-cni-config的configmap
找到cniBinDir更改成正確的路徑，從新生成全部pod

以上只列舉了bin目錄的錯誤，不一樣環境中也有多是cniConfDir的錯誤，修改成正確的就好。