docker-compose快速搭建 Prometheus+Grafana監控系統

1、說明
Prometheus負責收集數據，Grafana負責展現數據。其中採用Prometheus 中的 Exporter含：
1）Node Exporter，負責收集 host 硬件和操做系統數據。它將以容器方式運行在全部 host 上。
2）cAdvisor，負責收集容器數據。它將以容器方式運行在全部 host 上。
3）Alertmanager，負責告警。它將以容器方式運行在全部 host 上。

2、安裝docker，docker-compose
2.1 安裝docker
先安裝一個64位的Linux主機，其內核必須高於3.10，內存不低於1GB。在該主機上安裝Docker。

# 安裝依賴包
yum install -y yum-utils device-mapper-persistent-data lvm2
# 添加Docker軟件包源
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# 安裝Docker CE
yum install docker-ce -y
# 啓動
systemctl start docker
# 開機啓動
systemctl enable docker
# 查看Docker信息
docker info
2.2 安裝docker-compose

curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
3、添加配置文件

mkdir -p /usr/local/src/config
cd /usr/local/src/config
2.1 添加prometheus.yml配置文件，
vim prometheus.yml

# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ['192.168.159.129:9093']
# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "node_down.yml"
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['192.168.159.129:9090']

- job_name: 'cadvisor'
static_configs:
- targets: ['192.168.159.129:8080']

- job_name: 'node'
scrape_interval: 8s
static_configs:
- targets: ['192.168.159.129:9100']
2.2 添加郵件告警配置文件
添加配置文件alertmanager.yml，配置收發郵件郵箱
vim alertmanager.yml

global:
smtp_smarthost: 'smtp.163.com:25'　　#163服務器
smtp_from: 'tsiyuetian@163.com'　　　　　　　　#發郵件的郵箱
smtp_auth_username: 'tsiyuetian@163.com'　　#發郵件的郵箱用戶名，也就是你的郵箱
smtp_auth_password: 'TPP***'　　　　　　　　#發郵件的郵箱密碼
smtp_require_tls: false　　　　　　　　#不進行tls驗證

route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 10m
receiver: live-monitoring

receivers:
- name: 'live-monitoring'
email_configs:
- to: '1933306137@qq.com'　　　　　　　　#收郵件的郵箱
2.3 添加報警規則
添加一個node_down.yml爲 prometheus targets 監控
vim node_down.yml

groups:
- name: node_down
rules:
- alert: InstanceDown
expr: up == 0
for: 1m
labels:
user: test
annotations:
summary: "Instance {{ $labels.instance }} down"
description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."
4、編寫docker-compose
vim docker-compose-monitor.yml

version: '2'

networks:
monitor:
driver: bridge

services:
prometheus:
image: prom/prometheus
container_name: prometheus
hostname: prometheus
restart: always
volumes:
- /usr/local/src/config/prometheus.yml:/etc/prometheus/prometheus.yml
- /usr/local/src/config/node_down.yml:/etc/prometheus/node_down.yml
ports:
- "9090:9090"
networks:
- monitor

alertmanager:
image: prom/alertmanager
container_name: alertmanager
hostname: alertmanager
restart: always
volumes:
- /usr/local/src/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml
ports:
- "9093:9093"
networks:
- monitor

grafana:
image: grafana/grafana
container_name: grafana
hostname: grafana
restart: always
ports:
- "3000:3000"
networks:
- monitor

node-exporter:
image: quay.io/prometheus/node-exporter
container_name: node-exporter
hostname: node-exporter
restart: always
ports:
- "9100:9100"
networks:
- monitor

cadvisor:
image: google/cadvisor:latest
container_name: cadvisor
hostname: cadvisor
restart: always
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
ports:
- "8080:8080"
networks:
- monitor
5、啓動docker-compose

#啓動容器：
docker-compose -f /usr/local/src/config/docker-compose-monitor.yml up -d
#刪除容器：
docker-compose -f /usr/local/src/config/docker-compose-monitor.yml down
#重啓容器：
docker restart id
容器啓動以下：

prometheus targets界面以下：

備註：若是State爲Down，應該是防火牆問題，參考下面防火牆配置。

prometheus graph界面以下：

備註：若是沒有數據，同步下時間。

6、防火牆配置
6.1 關閉selinux

setenforce 0
vim /etc/sysconfig/selinux
6.2 配置iptables

#刪除自帶防火牆
systemctl stop firewalld.service
systemctl disable firewalld.service
#安裝iptables
yum install -y iptables-services
#配置
vim /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [24:11326]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9090 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3000 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9093 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9100 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
#啓動
systemctl restart iptables.service
systemctl enable iptables.service
7、配置Grafana
7.1 添加Prometheus數據源

2 配置dashboards

說明：能夠用自帶模板，也能夠去https://grafana.com/dashboards，下載對應的模板。

7.3 查看數據
我從網頁下載了docker相關的模板：Docker and system monitoring，893

輸入893，就會加載出下面的信息

導入後去首頁查看數據

8、附錄：單獨命令啓動各容器

#啓動prometheus
docker run -d -p 9090:9090 --name=prometheus \
-v /usr/local/src/config/prometheus.yml:/etc/prometheus/prometheus.yml \
-v /usr/local/src/config/node_down.yml:/etc/prometheus/node_down.yml \
prom/prometheus

# 啓動grafana
docker run -d -p 3000:3000 --name=grafana grafana/grafana

#啓動alertmanager容器
docker run -d -p 9093:9093 -v /usr/local/src/config/config.yml:/etc/alertmanager/config.yml --name alertmanager prom/alertmanager

#啓動node exporter
docker run -d \
-p 9100:9100 \
-v "/:/host:ro,rslave" \
--name=node_exporter \
quay.io/prometheus/node-exporter \
--path.rootfs /host

#啓動cadvisor
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest

---------------------