02-Kubenetes資源

時間 2019-11-17

標籤 kubenetes 資源简体版

原文原文鏈接

Kubenetes資源

經常使用資源對象

workload: Pod, ReplicaSet, Deployment, StatefulSet, DaemonSet, Job, Cronjob
負載均衡/服務發現：Service, Ingress, ...
配置與存儲： Volume， CSI
- cronfigMap，Secret
- DownwardAPI
集羣級別資源
- Namespace， node， role， ClusterRole， RoleBinding ， ClusterRoleBinding
元數據型資源
- HPA， PodTemplate， LimitRange

標籤labels

labels 與資源之間是多對多的關係java

標籤的定義通常從如下幾個角度定義node

版本：alpha beta canary stable
環境：dev pro qa
應用名稱
架構層級
分區標籤
品控標籤

標籤格式：mysql

key=value  
key: 字母 數字 _ .   
value：只能以字母數字開頭及結尾

經過標籤過濾nginx

kubectl get pods -l <labels>

查看全部標籤git

kubectl get pods --show-labels

打標籤github

kubectl label [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N
[--resource-version=version] [options]

標籤選擇器redis

等值關係：=， ==，!=sql
集合關係：docker

KEY in (VALUE1,VALUE2, ... )

KEY not in (VALUE1,VALUE2, ... )

!KEY * 不存在鍵

許多資源支持內嵌字段

matchLabels: 直接給定健值
matchExpressions: 基於給定的表達式來定義使用標籤選擇器，{key:"KEY", operator: "OPERATOR", values:[VAL1, VAL2, ...]}

操做符：In， NotIn, Exists, NotExists

建立資源的方式

apiserver僅接受JSON格式的資源定義；

yaml格式提供配置清單， apiserver可自動將其轉爲json格式，而後提交

大部分的資源的配置清單，主要5個一級資源

apiVersion
```
kubectl  api-versions
```
kind：資源類別
metadata: 元數據
- name
- namespace
- labels
- annotations
每一個資源的引用PATH 路徑

/api/GROUP/VERSION/namespaces/NAMESPACE_NAME/TYPE/NAME
spec
status

使用explain 查看定義

例如：

kubectl explain pods.metadata
kubectl explain pods.spec.containers

Pod

k8s管理的最小單位，一個pod中能夠有多個contaiers 例如

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    readinessProbe:
      httpGet:
        port: 80
      initialDelaySeconds: 2
      periodSeconds: 3
    livenessProbe:
      httpGet:
        port: 80
      initialDelaySeconds: 2
      periodSeconds: 3

  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ['/bin/sh','-c','ping','www.baidu.com']
  nodeSelector:
    kubernetes.io/hostname: 192.168.0.165

pods.spec.containers 必須

- name <string>
  image <string>
  imagePullPolicy     <string>  Always, Never, IfNotPresent. 
  * Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. (優化點)
 
  ports    <[]Object> 
  * 僅僅是說明性的
  - containerPort <integer> -required-
    hostIP   0.0.0.0
    hostPort  必須與containerPort 相同，大部分不須要定義該項
    name   名稱
    protocol 默認TCP

修改容器的啓動命令

command      <[]string>
args         <[]string>

- command 會覆蓋鏡像中的Entrypoint 與 command
- args 會覆蓋鏡像中的 command
  https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

nodeSelector <map [string]string>

節點選擇器，限定pod運行在哪些節點上。

使用標籤選擇器

nodeName<map [string]string>

直接選擇節點

annotations

註解，僅用於提供」元數據「並不提供資源兌現選擇。沒有大小限制。

restartPolicy

Always, OnFailure, Never Default to Always

hostNetwork

Host networking requested for this pod. Use the host's network namespace.If this option is set, the ports that will be used must be specified. Default to false.

pod直接使用主機的網絡名稱空間。有用但不經常使用，默認false。

pod的生命週期

串行執行多個 init_containters（初始化容器），初始化容器執行完成後退出。
啓動主容器 main containters
啓動後能夠執行 post start
主進程執行時能夠進行健康監測包括：liveness probe 與 readness probe
結束前能夠執行 pre stop

狀態：

Pending 等待調度，調度未完成
Running 運行狀態
Failed 失敗
Succeeded
Unknown

建立Pod：
apiServer etcd scheduler controller kubelet

容器重啓策略
restartPolicy

健康監測

健康監測主要針對容器，因此在 pod.spec.containers 層級下
監測類型

livenessProbe 存活性探測
readinessProbe 就緒性監測
lifecycle 容器啓動後或者中止前鉤子。

存活並不必定就緒
三種探針類型
ExecAction (exec)、TCPSocketAction (tcpSocket)、HTTPGetAction（httpGet）
健康監測主要參數

- exec  <Object> 使用命令監測 (重要)
- command   <[]string>
- httpGet 
- tcpSocket
- initialDelaySeconds (重要) 初始化等待時間
- periodSeconds (重要)  檢測間隔時間
- timeoutSeconds <integer> 錯誤超時時間 默認1秒
- failureThreshold  <integer>  最小失敗次數 默認3次
- successThreshold <integer>  最小成功次數 默認1次

lifecycle

容器啓動後或者中止前鉤子。

postStart
preStop
注意：lifecycle的postStart執行在容器command 以後。

FIELDS:

- exec          <Object>
- httpGet      <Object>   HTTPGet specifies the http request to perform.

env環境變量獲取

env不只能夠傳遞key value 的數據，還能夠從其餘地方傳值傳遞。
pods.spec.containers.env.valueFrom

- configMapKeyRef 
  Selects a key of a ConfigMap.
- fieldRef     <Object>
  Selects a field of the pod: supports metadata.name, metadata.namespace, metadata.labels, metadata.annotations, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP.
- resourceFieldRef     <Object>
  Selects a resource of the container: only resources limits and requests  (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.
- secretKeyRef <Object>
  Selects a key of a secret in the pod's namespace

pod 案例

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
    readinessProbe:
      httpGet:
        port: 80
      initialDelaySeconds: 2
      periodSeconds: 3
    livenessProbe:
      httpGet:
        port: 80
      initialDelaySeconds: 2
      periodSeconds: 3

  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: [ping, www.baidu.com]
  nodeSelector:
    kubernetes.io/hostname: 192.168.0.165

Pod控制器

ReplicaSet: 控制pod 副本數量，擴縮容機制
Deployment：ReplicaSet的控制器，滾動更新、回滾，聲明式定義。無狀態服務
DaemonSet: 確保每一個節點執行一個
Job : 執行一次
CronJob : 計劃任務
StatefuleSet：有狀態的服務
CDR： Custom Defined Resources
Operator

用戶應該直接操做Deployment。
最好不要將有狀態的服務部署在k8s上

deployment

更新策略
deployment.spec.strategy

Recreate
RollingUpdate
maxSurge Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%).
maxUnavailable Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%).
deployment.spec.revisionHistoryLimit
rc歷史保存數量

案例：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    author: huruizhi
    department: opreation
    usage: Java programs k8s template
  labels:
    module_name: pyfinance2v2-register-pro
    env: pro
    kind: deploy
  name: pyfinance2v2-register-pro
  namespace: default
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:    
      maxSurge: 2
      maxUnavailable: 2
  selector:
    matchLabels:
      module_name: pyfinance2v2-register-pro
      env: pro
      kind: pod
  template:
    metadata:
      creationTimestamp: null
      labels:
        module_name: pyfinance2v2-register-pro
        env: pro
        kind: pod
    spec:
      containers:
      - name: pyfinance2v2-register-pro
        image: harbor.pycf.com/pyfinance2v2/register:pro
        imagePullPolicy: Always
        ports:
        - containerPort: 5000 
        command: ['java','-jar','-Xms128m','-Xmx256m','/java8/app.jar','--server.port=5000']
        resources:
          limits:
            memory: 512Mi
          requests:
            memory: 128Mi
        env:
        - name: TZ
          value: Asia/Shanghai
        livenessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 40
          periodSeconds: 3
        readinessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 40
          periodSeconds: 3
          
      imagePullSecrets:
      - name: harborkey1
      restartPolicy: Always

DaemSet

在每一個節點上部署一個pod

支持滾動更新，支持兩種更新模式。可使用kubectl explain daemonset.spec.updateStrategy 查看。

手動更新 kubectl set image daemonset abc *=nginx:1.9.1

案例：

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filefeat-ds
  namespace: default
  labels:
        app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
      release: stable
  template:
    metadata:
      labels:
        app: filebeat
        release: stable
    spec:
      containers:
      - name: filefeat
        image: ikubenetes/filebeat:5.6.5-alpine
        env:
        - name: REDIS_HOST
          value: redis.default.svc.cluster.local
        - name: REDIS_LOG_LEVEL
          value: info

Service

Service的名稱解析依賴於dns 附件，網絡依賴於第三方網絡方案。

Service網絡是一個虛擬網絡，由kube-proxy維護。

工做模式：

iptables
ipvs

ipvs沒有被激活的狀況下自動使用iptables

iptables 查看：

iptables -L -n -t nat

svc.spec的重要字段

ClusterIP 通常不手動指定，能夠指定爲None 則爲無頭svc。

設置成無頭svc後 dns中的A記錄爲pod IP地址，A記錄的數量與pod數量至關

例如使用dig命令查看

# dig pyfinance2v2-register-pro.default.svc.cluster.local. @172.20.162.187 

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7_5.1 <<>> pyfinance2v2-register-pro.default.svc.cluster.local. @172.20.162.187
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3070
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pyfinance2v2-register-pro.default.svc.cluster.local. IN        A

;; ANSWER SECTION:
pyfinance2v2-register-pro.default.svc.cluster.local. 5 IN A 172.20.197.37
pyfinance2v2-register-pro.default.svc.cluster.local. 5 IN A 172.20.229.141
pyfinance2v2-register-pro.default.svc.cluster.local. 5 IN A 172.20.41.13

;; Query time: 2 msec
;; SERVER: 172.20.162.187#53(172.20.162.187)
;; WHEN: Wed Feb 13 10:23:49 CST 2019
;; MSG SIZE  rcvd: 281

ports <[]Object>
- port
- nodePort
- targetPort
selector
type ： ExternalName（訪問外部服務例如 GlusterFs）, ClusterIP, NodePort, and LoadBalancer( 外部負載均衡 ).
healthCheckNodePort
sessionAffinity ：ClientIP 和 None ，負載均衡調度策略。設置爲ClientIP 則將同一個ip的鏈接發送到後端同一個pod上。

域名後綴

默認爲svc_name.namespace_name.svc.cluster.local.

案例：

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert -f docker-compose-pro.yml
    kompose.version: 1.7.0 (HEAD)
  creationTimestamp: null
  labels:
    io.kompose.service: pyfinance2v2-amc-pro
  name: pyfinance2v2-amc-pro
  namespace: pyfinance2v2-pro
spec:
  type: NodePort
  ports:
  - name: "7562"
    port: 7562
    targetPort: 5000
    nodePort: 7562
  selector:
    io.kompose.service: pyfinance2v2-amc-pro
status:
  loadBalancer: {}

Ingress Controller

外部路由引入，7層負載均衡，能夠進行https 卸載。

HAproxy （不經常使用）
Nginx
Traefik https://docs.traefik.io/user-guide/kubernetes/
Envoy

案例：

http ingress: https://github.com/gjmzj/kubeasz/blob/master/docs/guide/ingress.md
https ingress: https://github.com/gjmzj/kubeasz/blob/master/docs/guide/ingress-tls.md

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-nginx-ingress
  namespace: default
spec:
  rules:
  - host: my-nginx.com
    http:
      paths:
      - path: /main
        backend:
          serviceName: my-nginx
          servicePort: 80
      - path: /busybox
        backend:
          serviceName: busybox-demo
          servicePort: 80

path: Path is an extended POSIX regex as defined by IEEE Std 1003.1, (i.e this follows the egrep/unix syntax, not the perl syntax) matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/'. If unspecified, the path defaults to a catch all sending traffic to the backend.

例如 path 設置爲 /main 則能夠訪問 /main /main1 等。不能訪問 / 、/aaa 等其餘路徑下資源

存儲卷管理

emptyDir 臨時存儲目錄
hostPath 主機存儲
網絡共享存儲： SAN NAS 分佈式存儲（glusterfs rbd cephfs ...）雲存儲

支持的存儲卷類型

kubectl explain pod.spec.volumes
kubectl explain persistentVolume.spec

定義一個簡單的emptyDir, 包涵兩個containers。兩個容器公用存儲卷。

apiVersion: v1
kind: Pod
metadata:
  name: busybox-demo
  labels:
    app: busybox
    role: volume_test
spec:
  containers:
  - name: httpd
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /usr/share/nginx/html/
      name: tmp-volume
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ['/bin/sh','-c','while true;do echo $(date) > /data/index.html;sleep 3;done']
    volumeMounts:
    - mountPath: /data/
      name: tmp-volume
  volumes:
  - name: tmp-volume
    emptyDir:
      sizeLimit: 200M

PV 與 PVC 資源

PV對象及主要參數

PV對象不屬於名稱空間

pv.Capacity

經過capacity給PV設置特定的大小。

pv.accessModes

k8s不會真正檢查存儲的訪問模式或根據訪問模式作訪問限制，只是對真實存儲的描述，最終的控制權在真實的存儲端。目前支持三種訪問模式：

* ReadWriteOnce – PV以 read-write 掛載到一個節點

* ReadOnlyMany – PV以read-only方式掛載到多個節點

* ReadWriteMany – PV以read-write方式掛載到多個節點

pv.spec.persistentVolumeReclaimPolicy

當前支持的回收策略:

* Retain – 容許用戶手動回收

* Recycle – 刪除PV上的數據 (「rm -rf /thevolume/*」)

* Delete – 刪除PV

PVC對象與重要參數

PVC 與PV對象關聯

pvc.spec.accessModes

同 pv對象

pvc.spec.resources

limits
requests

定義存儲大小的須要

案例 Glusterfs：

apiVersion: v1
kind: Endpoints
metadata:
  name: gfs-endpoint
  labels:
    storage: gfs
subsets:
- addresses:
  - ip: 192.168.0.165
  ports:
  - port: 49158
    protocol: TCP
- addresses:
  - ip: 192.168.0.162
  - ip: 192.168.0.166
  ports:
  - port: 49157
    protocol: TCP
--- 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gfs-pvc
spec:
  accessModes: 
  - ReadWriteMany
  volumeName: gfs-pv
  resources:
    requests:
      storage: 20Gi
---    
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gfs-pv
  labels:
    role: gfs-pv
spec:
  accessModes: 
  - ReadWriteMany
  glusterfs:  
    endpoints: gfs-endpoint
    path: gluster-test
  capacity:
    storage: 20Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gfs-pvc
spec:
  accessModes: 
  - ReadWriteMany
  volumeName: gfs-pv
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox-demo
  labels:
    app: busybox
    role: volume_test
spec:
  containers:
  - name: httpd
    image: nginx:latest
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /usr/share/nginx/html/busybox
      name: gfs-volume
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ['/bin/sh','-c','while true;do echo $(date) >> /data/index.html;sleep 3;done']
    volumeMounts:
    - mountPath: /data/
      name: gfs-volume
  volumes:
  - name: gfs-volume
    persistentVolumeClaim:
      claimName: gfs-pvc

StorageClass 動態生成pv

容器配置管理 secret 與 configmap

可使用環境變量以及掛載的方式配置到pod當中。

注意：環境變量的方式只能在容器啓動的時候注入，更新configmap 不會更新容器中環境變量的值。使用掛載的方式能夠實時更新。

建立configMap 有多種方式

使用kubectl create命令行方式

# Create a new configmap named my-config based on folder bar
  kubectl create configmap my-config --from-file=path/to/bar
  
  # Create a new configmap named my-config with specified keys instead of file basenames on disk
  kubectl create configmap my-config --from-file=key1=/path/to/bar/file1.txt --from-file=key2=/path/to/bar/file2.txt
  
  # Create a new configmap named my-config with key1=config1 and key2=config2
  kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2
  
  # Create a new configmap named my-config from the key=value pairs in the file
  kubectl create configmap my-config --from-file=path/to/bar
  
  # Create a new configmap named my-config from an env file
  kubectl create configmap my-config --from-env-file=path/to/bar.env

使用yaml文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-cfg
  namespace: default
data:
  cache_host: memcached-gcxt
  cache_port: "11211"
  cache_prefix: gcxt
  my.cnf: |
    [mysqld]
    log-bin = mysql-bin
  app.properties: |
    property.1 = value-1
    property.2 = value-2
    property.3 = value-3

使用命令行建立更靈活。

可使用inotify監控配置文件實現重載

例如:

#!/bin/sh
oldcksum=`cksum /etc/nginx/conf.d/default.conf`

inotifywait -e modify,move,create,delete -mr --timefmt '%d/%m/%y %H:%M' --format '%T' \
/etc/nginx/conf.d/ | while read date time; do

    newcksum=`cksum /etc/nginx/conf.d/default.conf`
    if [ "$newcksum" != "$oldcksum" ]; then
        echo "At ${time} on ${date}, config file update detected."
        oldcksum=$newcksum
        nginx -s reload
    fi

done

關於configmap的詳細總結： https://www.cnblogs.com/breezey/p/6582082.html

StatefuleSet

特色：

穩定且惟一的網絡標識符；
穩定且持久的存儲；
有序、平滑的部署和擴展；
有序、平滑的刪除和終止；
有序的滾動更新；

三個主要組件：headless service 、 StatefulSet、 volumeClaimTemplate

名稱解析：

pod_name,service_name.ns_name.svc.cluster.local

更新策略

sts.spec.updateStrategy.rollingUpdate

partition 定義更新的邊界，例如定義爲3 則編號 >=3的 pod會更新，模擬金絲雀發佈

PV定義

apiVersion: v1
kind: Endpoints
metadata:
  name: gfs-endpoint
  labels:
    storage: gfs
subsets:
- addresses:
  - ip: 192.168.0.165
  ports:
  - port: 49158
    protocol: TCP
- addresses:
  - ip: 192.168.0.162
  - ip: 192.168.0.166
  ports:
  - port: 49157
    protocol: TCP

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: gfs-pv-01
  labels:
    role: gfs-pv-01
spec:
  accessModes: 
  - ReadWriteMany
  - ReadWriteOnce
  glusterfs:  
    endpoints: gfs-endpoint
    path: pv-01
  capacity:
    storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gfs-pv-02
  labels:
    role: gfs-pv-02
spec:
  accessModes:
  - ReadWriteMany
  - ReadWriteOnce
  glusterfs:
    endpoints: gfs-endpoint
    path: pv-02
  capacity:
    storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gfs-pv-03
  labels:
    role: gfs-pv-03
spec:
  accessModes:
  - ReadWriteMany
  - ReadWriteOnce
  glusterfs:
    endpoints: gfs-endpoint
    path: pv-03
  capacity:
    storage: 5Gi
--- 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gfs-pv-04
  labels:
    role: gfs-pv-04
spec:
  accessModes:
  - ReadWriteMany
  - ReadWriteOnce
  glusterfs:
    endpoints: gfs-endpoint
    path: pv-04
  capacity:
    storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gfs-pv-05
  labels:
    role: gfs-pv-05
spec:
  accessModes:
  - ReadWriteMany
  - ReadWriteOnce
  glusterfs:
    endpoints: gfs-endpoint
    path: pv-05
  capacity:
    storage: 5Gi

StatefulSet定義

apiVersion: v1
kind: Service
metadata:
  name: myapp-svc
  labels:
    roles: myapp-svc-test
spec:
  clusterIP: None
  ports:
  - targetPort: 80
    port: 80
  selector:
    roles: myapp-pod
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp-sts
  labels:
    roles: myapp-sts-test
spec:
  replicas: 3
  serviceName: myapp-svc
  selector: 
    matchLabels:
      roles: myapp-pod
  template:
    metadata:
      labels:
        roles: myapp-pod
    spec:
      containers:
       - name: httpd
         image: nginx:latest
         imagePullPolicy: IfNotPresent
         volumeMounts:
         - mountPath: /usr/share/nginx/html/busybox
           name: gfs-volume
  volumeClaimTemplates:
  - metadata:
      name: gfs-volume
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  updateStrategy:
    rollingUpdate: 
      partition: 2