k8s上使用prometheus監控websocket服務

說明

咱們當前k8s集羣上使用prometheus作監控，因爲開發同窗有部分業務使用websocket接口，也爲了能有效對業務應用進行監控和報警，頗有必要對websocket api接口存活性進行探測和監管。具體方案、實施流程和測試詳見下文。

部署簡單 websocket service

咱們定義一個簡單的websocket service用來監控報警測試，以下：

# 建立虛擬環境，也能夠直接在宿主機上部署
mkvirtualenv -p /usr/bin/python3 websocket-server
# 安裝必要包
pip3 install websockets

# cat websocket-server.py

import asyncio
import websockets

async def echo(websocket, path):
    async for message in websocket:
        message = "I got your message: {}".format(message)
        await websocket.send(message)
# 定義的ip地址要能與k8s通訊
asyncio.get_event_loop().run_until_complete(websockets.serve(echo, '192.168.128.6', 8765))
asyncio.get_event_loop().run_forever()

# 啓動websocket服務
python websocket-server.py &
# 查看服務
netstat -lnp|grep 8765

websocket-exports

這裏咱們定義一個deployment用來將監控的多個websocket apimetrics對接到prometheus，內容以下：

k8s websocket deployment

# cat websocket-kube-mon-prometheus.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: wss
    app.kubernetes.io/version: v1.8.0
  name: websocket-exporter
  namespace: kube-mon
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: wss
  template:
    metadata:
      labels:
        app.kubernetes.io/name: wss
        app.kubernetes.io/version: v1.8.0
    spec:
      containers:
      - image: registry.cn-shanghai.aliyuncs.com/ai-voice-test/wss-expoter:v0.0.1
        env:
          - name: ENDPOINT
            #多個ws用逗號分開
            value: ws://www.abc.com,ws://192.168.128.6:8765
        name: websocket-exporter
        ports:
        - containerPort: 9189
          name: wss-metrics

k8s websocket service

定義websocket service用來被prometheus監控，內容以下：

# cat service-websocket.yaml

apiVersion: v1
kind: Service
metadata:
  name: websocket
  namespace: kube-mon
spec:
  # 暫使用nodeport的形式
  type: NodePort
  ports:
  - port: 9189
    targetPort: 9189
    protocol: TCP
    nodePort: 32071
  selector:
    app.kubernetes.io/name: wss

獲取ip port

# 啓動上面deploy和service
kubectl apply -f websocket-kube-mon-prometheus.yaml 
kubectl apply -f service-websocket.yaml
# 查看pod和service
kubectl get pod -n kube-mon
kubectl get svc -n kube-mon
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
websocket              NodePort    192.168.237.56    <none>        9189:32071/TCP   1h

配置prometheus

配置prometheus監控

# vim sidecar/cm-kube-mon-sidecar.yaml //添加如下配置
    - job_name: 'websocket'
      static_configs:
        - targets: ['192.168.237.56:9189']

# 重載
kubectl apply -f sidecar/cm-kube-mon-sidecar.yaml 
# prometheus reload：
curl -X POST http://prometheus-pod-ip:9090/-/reload

配置告警規則

# vim sidecar/rules-cm-kube-mon-sidecar.yaml //添加如下配置
      - alert: websocket 接口探測到異常
        expr: websocket{job="websocket"} < 1
        for: 30s
        labels:
          severity: 緊急
        annotations:
          #summary: "接口{{ $labels.url }} 探測異常"
          description: "websocket地址: {{ $labels.url }} 探測異常 , 狀態爲: down ."

# 重載，prometheus有熱更新，稍等待1分鐘左右便可
kubectl apply -f sidecar/rules-cm-kube-mon-sidecar.yaml

報警測試

關閉測試websocket service

# 查看進程號
netstat -lnp|grep 8765
# 殺掉進程
kill you-id

咱們能夠終端請求直接看到接口監控狀態，以下：

curl 192.168.237.56:9189/metrics
# HELP websocket websocket_help
# TYPE websocket gauge
websocket{url="ws://192.168.128.6:8765"} 0

稍等待一下子，報警信息報出，內容以下：

從新運行websocket service

python websocket-server.py &

curl 192.168.237.56:9189/metrics
# HELP websocket websocket_help
# TYPE websocket gauge
websocket{url="ws://192.168.128.6:8765"} 1

稍等待一下子，恢復信息報出，內容以下：

參考庫

• websocket_exporter