Prometheus 應用監控

本文講述如何實現Prometheus的應用監控

監控Nginx

nginx：1.16.一、nginx-vts-exporter：0.10.三、nginx-module-vts：1.6.3

監控Nginx主要用到如下三個模塊：

• nginx-module-vts：Nginx的監控模塊，可以提供JSON格式的數據產出。
• nginx-vts-exporter：主要用於收集Nginx的監控數據，並給Prometheus提供監控接口，默認端口號9913。
• Prometheus：監控Nginx-vts-exporter提供的Nginx數據，並存儲在時序數據庫中，可使用PromQL對時序數據進行查詢和聚合

nginx-module-vts模塊編譯
nginx_vts_exporter依賴nginx-module-vts模塊，安裝此模塊無需任何其餘依賴。模塊與Nginx的版本兼容性以下：

1.11.x (last tested: 1.11.10)
1.10.x (last tested: 1.10.3)
1.8.x (last tested: 1.8.0)
1.6.x (last tested: 1.6.3)
1.4.x (last tested: 1.4.7)

1.安裝依賴包

$ yum -y install gcc gcc-c++ autoconf automake make      
$ yum -y install zlib zlib-devel openssl openssl-devel pcre pcre-devel

2.添加www用戶

$ groupadd -f www
$ useradd -g www www

3.下載nginx（版本1.16.1）

$ wget http://nginx.org/download/nginx-1.16.1.tar.gz
$ tar xf nginx-1.16.1.tar.gz

3.克隆vts模塊倉庫

https://github.com/vozlt/nginx-module-vts/releases

$ git clone git://github.com/vozlt/nginx-module-vts.git

4.編譯nginx配置，添加vts模塊
在nginx編譯時添加vts模塊：--add-module=/path/to/nginx-module-vts

$ cd nginx-1.16.1/

# 配置nginx
# --prefix 指定安裝的目錄
# /usr/local/nginx 是安裝目錄，不能和本身下載的文件目錄重了
#./configure --prefix=/usr/local/nginx

$ ./configure --user=www --group=www \
--prefix=/usr/local/nginx \
--with-http_stub_status_module \
--with-http_ssl_module \
--with-stream \
--with-http_gzip_static_module \
--with-http_sub_module \
--add-module=/root/nginx-module-vts

$ make && make install

5.Nginx config配置
更改Nginx Conf的配置，添加監控接口/status

$ vim /usr/local/nginx/conf/nginx.conf
user www;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  65;

    # enabled nginx status
    vhost_traffic_status_zone;
    gzip  on;

    server {
        listen       80;
        server_name  localhost;
        location /status {
             vhost_traffic_status_display;
             vhost_traffic_status_display_format html;
             # Only allow local access the url
             allow 127.0.0.1;
        }
      }
}

檢查nginx語法並啓動

$ /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

$ /usr/local/nginx/sbin/nginx

6.curl查看捕獲nginx數據

$ curl http://127.0.0.1/status/format/json

7.安裝nginx-vts-exporter
下載地址：https://github.com/hnlq715/nginx-vts-exporter/releases

$ tar xf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
$ cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter  /usr/local/bin/

//啓動命令
nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1/status/format/json

6.將nginx-vts-exporter添加system系統服務

$ vim /usr/lib/systemd/system/nginx-vts-exporter.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/nginx-vts-exporter \
-nginx.scrape_uri=http://127.0.0.1/status/format/json

[Install]
WantedBy=multi-user.target

7.啓動添加的system服務

$ systemctl daemon-reload
$ systemctl start nginx-vts-exporter.service
$ systemctl enable nginx-vts-exporter.service

// nginx-vts-exporter 默認端口 - 9913
$ netstat -lntup | grep "9913"
tcp6       0      0 :::9913                 :::*                    LISTEN      9195/nginx-vts-expo

8.使用prometheus監控修改監控端配置文件

$ vim /usr/local/prometheus/prometheus.yml
  - job_name: 'nginx'
    static_configs:
      - targets: ['10.0.0.171:9913']

9.啓動prometheus服務

$ systemctl restart prometheus.service

10.查看監控端是否介入

11.Granfana導入Nginx監控圖表

https://grafana.com/grafana/dashboards/2949

監控MySQL
文中主要監控MySQL/MySQL主從信息
版本：mysql-5.7，mysql_exporter-0.12.1

mysql_exporter下載地址：https://github.com/prometheus/mysqld_exporter/releases

一、下載mysql_exporter並解壓

$ tar xf mysqld_exporter-0.12.1.linux-amd64.tar.gz

// 將mysql_exporter二進制文件拷貝至/usr/local/bin
$ cp mysqld_exporter-0.12.1.linux-amd64/mysqld_exporter /usr/local/bin/

2.須要受權用戶給exporter使用

> CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'abc12345' WITH MAX_USER_CONNECTIONS 3;

// 可查看主從運行狀況查看線程，及全部數據庫。
> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

爲該用戶設置最大鏈接數爲了不監控數據過大致使服務器超載

3.修改mysql配置文件，添加剛纔建立的exporter用戶和密碼

$ vim /etc/my.cnf
[client]
user=exporter
password=abc12345

4.啓動exporter客戶端，需指定mysql配置文件，讀取exporter用戶和密碼

$ mysqld_exporter --config.my-cnf=/etc/my.cnf

經常使用參數：
// 選擇採集innodb
--collect.info_schema.innodb_cmp
// innodb存儲引擎狀態
--collect.engine_innodb_status
// 指定配置文件
--config.my-cnf="/etc/my.cnf"

5.添加system系統服務

$ vim /usr/lib/systemd/system/mysql_exporter.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/mysqld_exporter \
--config.my-cnf=/etc/my.cnf

[Install]
WantedBy=multi-user.target

6.啓動添加的system服務

$ systemctl daemon-reload
$ systemctl start mysql_exporter.service
$ systemctl enable mysql_exporter.service

// mysql_export默認端口 - 9104
$ netstat -lntup | grep "9104"
tcp6       0      0 :::9104                 :::*                    LISTEN      34137/mysqld_export

7.curl查看捕獲mysql數據

curl http://localhost:9104/metrics

8.使用prometheus監控修改監控端配置文件

$ vim /usr/local/prometheus/prometheus.yml
  - job_name: 'mysql'
    scrape_interval: 5s
    # 靜態添加node
    static_configs:
      - targets: ['10.0.0.171:9104']

9.啓動prometheus服務

$ systemctl restart prometheus.service

10.查看監控端是否介入

11.Granfana導入MySQL監控圖表

https://grafana.com/grafana/dashboards/8564/revisions

監控Redis
本文主要監控redis信息
版本：redis-3.2.2，redis_exporter-1.3.4

安裝部署redis：https://blog.51cto.com/12643266/2349586
redis_exporter下載地址：https://github.com/oliver006/redis_exporter/releases/

1.下載redis_exporter插件

代理插件不必定非要安裝在redis端

$ wget https://github.com/oliver006/redis_exporter/releases/download/v1.3.4/redis_exporter-v1.3.4.linux-amd64.tar.gz
$ tar xf redis_exporter-v1.3.4.linux-amd64.tar.gz
$ cp redis_exporter-v1.3.4.linux-amd64/redis_exporter  /usr/local/bin/

2.啓動redis_exporter登錄redis

// 無密碼
redis_exporter -redis.addr 127.0.0.1:6379 
// 有密碼
redis_exporter -redis.addr 172.0.0.1:6379  -redis.password 123456

3.添加system系統服務

$ vim /usr/lib/systemd/system/redis_exporter.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/redis_exporter \
-redis.addr 127.0.0.1:6379 

[Install]
WantedBy=multi-user.target

4.啓動添加的system服務

$ systemctl daemon-reload
$ systemctl start redis_exporter.service
$ systemctl enable redis_exporter.service

// redis_exporter默認端口 - 9121
$ netstat -lntup | grep 9121""
tcp6       0      0 :::9121                 :::*                    LISTEN      4465/redis_exporter

5.本地curl訪問

curl http://127.0.0.1:9121/metrics

6.prometheus.yml加入redis節點，而後重啓prometheus：

- job_name: 'redis'
    static_configs:
      - targets: ['10.0.0.171:9121']

7.重啓prometheus服務端

systemctl restart prometheus.service

8.查看監控端是否介入

9.Granfana導入Redis監控圖表
redis節點監控：https://grafana.com/grafana/dashboards/2751
redis哨兵監控：https://grafana.com/grafana/dashboards/9570