MetalLB模擬私有云K8s Service LoadBalancer模式

時間 2020-07-09

標籤 metallb 模擬私有 k8s service loadbalancer 模式简体版

原文原文鏈接

Kubernetes Service 有3種暴露服務到外網的方式:node

NodePort 模式
Ingress 模式
LoadBalancer 模式

其中 NodePort 和 Ingress 模式都可在測試環境模擬, 可是 LoadBalancer 模式依賴於雲廠商. 若是是在本地或者私有云中部署試用Kubernetes, 那麼將沒法直接使用.linux

MetalLB 提供了這樣一種模擬環境. 他嵌入 k8s 中, 容許創建 LoadBalancer模式的 Service. 下面作一個簡單 demo.git

一. 安裝

1.1 準備工做

首先必須已經成功安裝了 kubernetes 集羣. github

其次, 若是 kube-proxy 使用了 IPVS 方式, 那麼須要開啓嚴格 ARP 模式.express

1.2 安裝

應用如下 Manifest:api

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

以上文件 namespace.yaml 和 metallb.yaml 在牆內沒法直接打開和下載, 可見文末附錄部分.bash

準備和安裝部分更多細節參考官方文檔.網絡

二. 配置

網絡配置支持Layer 2 和 BGP, 測試環境通常使用 Layer 2. 支持2種地址池配置方式:session

掩碼方式: NN.NN.NN.NN/NN
地址範圍: NN.NN.NN.NN-NN.NN.NN.NN

具體的地址範圍根據 k8s 集羣所在的網絡決定. 好比集羣所在局域網 IP 範圍爲 192.168.31.255/24, 那麼能夠設置爲其中一部分 192.168.31.254/28 (或 192.168.31.240-192.168.31.254)app

本地新建 metallb-l2-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.31.254/28

應用該配置:

kubectl apply -f metallb-l2-config.yaml

更多有關網絡配置相關信息可參考官方文檔和官方示例 (如沒法打開, 見附錄3).

三. 附錄

3.1 MetalLB 的安裝配置文件

1) namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    app: metallb

2) metallb.yaml

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities: []
  allowedHostPaths: []
  defaultAddCapabilities: []
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  hostIPC: false
  hostNetwork: false
  hostPID: false
  privileged: false
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - secret
  - emptyDir
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  labels:
    app: metallb
  name: speaker
  namespace: metallb-system
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
  - NET_ADMIN
  - NET_RAW
  - SYS_ADMIN
  allowedHostPaths: []
  defaultAddCapabilities: []
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    rule: RunAsAny
  hostIPC: false
  hostNetwork: true
  hostPID: false
  hostPorts:
  - max: 7472
    min: 7472
  privileged: true
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - secret
  - emptyDir
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: metallb
  name: speaker
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: metallb
  name: metallb-system:controller
rules:
- apiGroups:
  - ''
  resources:
  - services
  verbs:
  - get
  - list
  - watch
  - update
- apiGroups:
  - ''
  resources:
  - services/status
  verbs:
  - update
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resourceNames:
  - controller
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: metallb
  name: metallb-system:speaker
rules:
- apiGroups:
  - ''
  resources:
  - services
  - endpoints
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resourceNames:
  - speaker
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: config-watcher
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: pod-lister
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: metallb
  name: metallb-system:controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:controller
subjects:
- kind: ServiceAccount
  name: controller
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: metallb
  name: metallb-system:speaker
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:speaker
subjects:
- kind: ServiceAccount
  name: speaker
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: config-watcher
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: config-watcher
subjects:
- kind: ServiceAccount
  name: controller
- kind: ServiceAccount
  name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: pod-lister
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-lister
subjects:
- kind: ServiceAccount
  name: speaker
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: metallb
    component: speaker
  name: speaker
  namespace: metallb-system
spec:
  selector:
    matchLabels:
      app: metallb
      component: speaker
  template:
    metadata:
      annotations:
        prometheus.io/port: '7472'
        prometheus.io/scrape: 'true'
      labels:
        app: metallb
        component: speaker
    spec:
      containers:
      - args:
        - --port=7472
        - --config=config
        env:
        - name: METALLB_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: METALLB_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: METALLB_ML_BIND_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: METALLB_ML_LABELS
          value: "app=metallb,component=speaker"
        - name: METALLB_ML_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: METALLB_ML_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: memberlist
              key: secretkey
        image: metallb/speaker:v0.9.3
        imagePullPolicy: Always
        name: speaker
        ports:
        - containerPort: 7472
          name: monitoring
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
            - SYS_ADMIN
            drop:
            - ALL
          readOnlyRootFilesystem: true
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/os: linux
      serviceAccountName: speaker
      terminationGracePeriodSeconds: 2
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: metallb
    component: controller
  name: controller
  namespace: metallb-system
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: metallb
      component: controller
  template:
    metadata:
      annotations:
        prometheus.io/port: '7472'
        prometheus.io/scrape: 'true'
      labels:
        app: metallb
        component: controller
    spec:
      containers:
      - args:
        - --port=7472
        - --config=config
        image: metallb/controller:v0.9.3
        imagePullPolicy: Always
        name: controller
        ports:
        - containerPort: 7472
          name: monitoring
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true
      nodeSelector:
        beta.kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: controller
      terminationGracePeriodSeconds: 0

3) example-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    # The peers section tells MetalLB what BGP routers to connect too. There
    # is one entry for each router you want to peer with.
    peers:
    - # The target IP address for the BGP session.
      peer-address: 10.0.0.1
      # The BGP AS number that MetalLB expects to see advertised by
      # the router.
      peer-asn: 64512
      # The BGP AS number that MetalLB should speak as.
      my-asn: 64512
      # (optional) the TCP port to talk to. Defaults to 179, you shouldn't
      # need to set this in production.
      peer-port: 179
      # (optional) The proposed value of the BGP Hold Time timer. Refer to
      # BGP reference material to understand what setting this implies.
      hold-time: 120s
      # (optional) The router ID to use when connecting to this peer. Defaults
      # to the node IP address. Generally only useful when you need to peer with
      # another BGP router running on the same machine as MetalLB.
      router-id: 1.2.3.4
      # (optional) Password for TCPMD5 authenticated BGP sessions
      # offered by some peers.
      password: "yourPassword"
      # (optional) The nodes that should connect to this peer. A node
      # matches if at least one of the node selectors matches. Within
      # one selector, a node matches if all the matchers are
      # satisfied. The semantics of each selector are the same as the
      # label- and set-based selectors in Kubernetes, documented at
      # https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/.
      # By default, all nodes are selected.
      node-selectors:
      - # Match by label=value
        match-labels:
          kubernetes.io/hostname: prod-01
        # Match by 'key OP values' expressions
        match-expressions:
        - key: beta.kubernetes.io/arch
          operator: In
          values: [amd64, arm]

    # The address-pools section lists the IP addresses that MetalLB is
    # allowed to allocate, along with settings for how to advertise
    # those addresses over BGP once assigned. You can have as many
    # address pools as you want.
    address-pools:
    - # A name for the address pool. Services can request allocation
      # from a specific address pool using this name, by listing this
      # name under the 'metallb.universe.tf/address-pool' annotation.
      name: my-ip-space
      # Protocol can be used to select how the announcement is done.
      # Supported values are bgp and layer2.
      protocol: bgp
      
      # A list of IP address ranges over which MetalLB has
      # authority. You can list multiple ranges in a single pool, they
      # will all share the same settings. Each range can be either a
      # CIDR prefix, or an explicit start-end range of IPs.
      addresses:
      - 198.51.100.0/24
      - 192.168.0.150-192.168.0.200
      # (optional) If true, MetalLB will not allocate any address that
      # ends in .0 or .255. Some old, buggy consumer devices
      # mistakenly block traffic to such addresses under the guise of
      # smurf protection. Such devices have become fairly rare, but
      # the option is here if you encounter serving issues.
      avoid-buggy-ips: true
      # (optional, default true) If false, MetalLB will not automatically
      # allocate any address in this pool. Addresses can still explicitly
      # be requested via loadBalancerIP or the address-pool annotation.
      auto-assign: false
      # (optional) A list of BGP advertisements to make, when
      # protocol=bgp. Each address that gets assigned out of this pool
      # will turn into this many advertisements. For most simple
      # setups, you'll probably just want one.
      #
      # The default value for this field is a single advertisement with
      # all parameters set to their respective defaults.
      bgp-advertisements:
      - # (optional) How much you want to aggregate up the IP address
        # before advertising. For example, advertising 1.2.3.4 with
        # aggregation-length=24 would end up advertising 1.2.3.0/24.
        # For the majority of setups, you'll want to keep this at the
        # default of 32, which advertises the entire IP address
        # unmodified.
        aggregation-length: 32
        # (optional) The value of the BGP "local preference" attribute
        # for this advertisement. Only used with IBGP peers,
        # i.e. peers where peer-asn is the same as my-asn.
        localpref: 100
        # (optional) BGP communities to attach to this
        # advertisement. Communities are given in the standard
        # two-part form <asn>:<community number>. You can also use
        # alias names (see below).
        communities:
        - 64512:1
        - no-export
    # (optional) BGP community aliases. Instead of using hard to
    # read BGP community numbers in address pool advertisement
    # configurations, you can define alias names here and use those
    # elsewhere in the configuration. The "no-export" community used
    # above is defined below.
    bgp-communities:
      # no-export is a well-known BGP community that prevents
      # re-advertisement outside of the immediate autonomous system,
      # but people don't usually recognize its numerical value. :)
      no-export: 65535:65281

3.2 參考

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。