Centos7.5 Prometheus2.5配置和基於Consul1.2.4的服務發現

1、Prometheus安裝及配置

請參考：
CentOS7.5 Prometheus2.5+Grafana5.4監控部署

二 、基於Consul的服務發現

一、概述

• Consul 是一個支持多數據中心分佈式高可用的服務發現和配置共享的服務軟件.
• Consul 由 HashiCorp公司用Go語言開發, 基於Mozilla Public License 2.0的協議進行開源. 
• Consul 支持健康檢查,並容許 HTTP 和 DNS 協議調用 API 存儲鍵值對.
• 命令行超級好用的虛擬機管理軟件 vgrant 也是 HashiCorp 公司開發的產品.
• 一致性協議採用 Raft 算法,用來保證服務的高可用. 使用 GOSSIP 協議管理成員和廣播消息, 而且支持 ACL 訪問控制.

架構圖

二、下載及安裝

wget https://releases.hashicorp.com/consul/1.2.4/consul_1.2.4_linux_amd64.zip
unzip consul_1.2.4_linux_amd64.zip -d /app/prometheus/bin/
cd /app/prometheus/bin/
chown -R prometheus.prometheus consul

三、建立Consul.service 的 systemd unit 文件

# vim /usr/lib/systemd/system/consul.service
[Unit]
Description=consul
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/app/prometheus/bin/consul agent \
-server -bootstrap-expect 1 \
-bind=0.0.0.0 \
-client=172.16.9.201 \
-data-dir=/app/prometheus/consuld/data/consul \
-node=172.17.9.201 \
-config-dir=/app/prometheus/consuld/conf \
-ui
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

參數說明

• –net=host docker參數, 使得docker容器越過了net namespace的隔離，免去手動指定端口映射的步驟
• -server consul支持以server或client的模式運行, server是服務發現模塊的核心, client主要用於轉發請求
• -advertise 將本機私有IP傳遞到consul
• -bootstrap-expect 指定consul將等待幾個節點連通，成爲一個完整的集羣
• -retry-join 指定要加入的consul節點地址，失敗會重試, 可屢次指定不一樣的地址
• -client consul綁定在哪一個client地址上，這個地址提供HTTP、DNS、RPC等服務，默認是127.0.0.1
• -bind 該地址用來在集羣內部的通信，集羣內的全部節點到地址都必須是可達的，默認是0.0.0.0
• allow_stale 設置爲true, 代表能夠從consul集羣的任一server節點獲取dns信息, false則代表每次請求都會通過consul server leader

四、啓動服務

systemctl daemon-reload
 systemctl start consul.service 
 systemctl enable consul.service

五、查看運行狀態

# systemctl status consul.service 
● consul.service - consul
   Loaded: loaded (/usr/lib/systemd/system/consul.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2018-12-11 15:23:12 CST; 20min ago
     Docs: https://prometheus.io/
 Main PID: 5721 (consul)
   CGroup: /system.slice/consul.service
           └─5721 /app/prometheus/bin/consul agent -server -bootstrap-expect 1 -bind=0.0.0.0 -client=172.16.9.201 -data-dir=/app/prometheus/consuld/data/consul -node=172.17.9.201 -config...

12月 11 15:23:18 prometheus-node1 consul[5721]: 2018/12/11 15:23:18 [INFO] raft: Election won. Tally: 1
12月 11 15:23:18 prometheus-node1 consul[5721]: 2018/12/11 15:23:18 [INFO] raft: Node at 172.16.9.201:8300 [Leader] entering Leader state
12月 11 15:23:18 prometheus-node1 consul[5721]: 2018/12/11 15:23:18 [INFO] consul: cluster leadership acquired
12月 11 15:23:18 prometheus-node1 consul[5721]: 2018/12/11 15:23:18 [INFO] consul: New leader elected: 172.17.9.201
12月 11 15:23:18 prometheus-node1 consul[5721]: 2018/12/11 15:23:18 [INFO] consul: member '172.17.9.201' joined, marking health alive
12月 11 15:23:18 prometheus-node1 consul[5721]: 2018/12/11 15:23:18 [INFO] agent: Synced node info
12月 11 15:23:25 prometheus-node1 consul[5721]: ==> Newer Consul version available: 1.4.0 (currently running: 1.2.4)
12月 11 15:40:21 prometheus-node1 consul[5721]: 2018/12/11 15:40:21 [WARN] agent: Service name "node_exporter" will not be discoverable via DNS due to invalid characters. Val...and dashes.
12月 11 15:40:21 prometheus-node1 consul[5721]: 2018/12/11 15:40:21 [INFO] agent: Synced service "node_exporter"
12月 11 15:42:08 prometheus-node1 consul[5721]: 2018/12/11 15:42:08 [INFO] agent: Synced service "node_exporter"
Hint: Some lines were ellipsized, use -l to show in full.

http://172.16.9.201:8500/ui/

六、配置Consul.service自動註冊

yum -y install jq

服務查詢

# curl -s http://172.16.9.201:8500/v1/catalog/services|jq
{
  "consul": []
}

使用HTTP接口服務註冊：

# curl -X PUT -d '{"ID": "node_exporter", "Name": "node_exporter", "Address": "172.16.9.201", "Port": 9100, "Tags": ["lock"], "EnableTagOverride": false}' http://172.16.9.201:8500/v1/agent/service/register
# curl -s http://172.16.9.201:8500/v1/catalog/services|jq
{
  "consul": [],
  "node_exporter": [
    "lock"
  ]
}

七、prometheus服務配置文件

# vim prometheus.yml 

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 172.16.9.201:9093
      - 172.16.9.202:9093

rule_files:
   - /app/prometheus/cfg/rule.yml
scrape_configs:
  - job_name: 'prometheus'
    metrics_path:    /metrics
    honor_labels:    false
    static_configs:
      - targets: ['localhost:9090']
        labels:
          group: 'node'
          service: 'prometheus'
  - job_name: 'prod_discover'
    metrics_path: /metrics
    honor_labels: false
    consul_sd_configs:
    - server: '172.16.9.201:8500'
      services: ['node_exporter']
      tag_separator: ''
    relabel_configs:
    - source_labels: ['__meta_consul_tags']
      target_label: 'product'
    - source_labels: ['__meta_consul_dc']
      target_label: 'idc'
    - source_labels: ['__meta_consul_service']
      target_label: 'service'
    - source_labels: ['job']
      target_label: 'environment'
      regex:        '(.*)_discover'
      replacement:   '${1}'

打開WEB界面

http://172.16.9.201:9090/targets