Prometheus + AlertManager 郵件報警

安裝

wget https://github.com/prometheus/alertmanager/releases/download/v0.13.0/alertmanager-0.13.0.linux-amd64.tar.gz

tar -axvf alertmanager-0.13.0.linux-amd64.tar.gz

配置AlertManager

AlertManager安裝目錄下有默認的simple.yml文件，能夠建立新的配置文件，在啓動時指定便可。

配置文件以下：

global:
  smtp_smarthost: 'smtp.163.com:25'
  smtp_from: 'jugglee@163.com'
  smtp_auth_username: 'jugglee@163.com'
  smtp_auth_password: 'admin123'
  smtp_require_tls: false

templates:
  - '/alertmanager/template/*.tmpl'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 10m
  receiver: default-receiver

receivers:
- name: 'default-receiver'
  email_configs:
  - to: 'whiiip@163.com'
    html: '{{ template "alert.html" . }}'
    headers: { Subject: "[WARN] 報警郵件test" }

smtp_smarthost是用於發送郵件的郵箱的SMTP服務器地址+端口

smtp_auth_password是發送郵箱的受權碼而不是登陸密碼

smtp_require_tls不設置的話默認爲true，當爲true時會有starttls錯誤，能夠用其餘辦法解決。爲了簡單這裏直接設置爲false

templates指出郵件的模板路徑

receivers下html指出郵件內容模板名，這裏模板名爲「alert.html」，在模板路徑中的某個文件中定義

headers爲郵件標題

配置Prometheus報警規則

配置rule.yml

groups:
- name: test-rule
  rules:
  - alert: clients
    expr: redis_connected_clients > 1
    for: 1m
    labels:
      severity: warning 
    annotations:
      summary: "{{$labels.instance}}: Too many clients detected"
      description: "{{$labels.instance}}: Client num is above 80% (current value is: {{ $value }}"

在prometheus.yml中指定rule.yml的路徑

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ["localhost:9093"]

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - /rule.yml
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: redis_exporter

    static_configs:
      - targets: ['localhost:9122']

編寫郵件模板

文件後綴爲tmpl

{{ define "alert.html" }}
<table>
    <tr><td>報警名</td><td>開始時間</td></tr>
    {{ range $i, $alert := .Alerts }}
        <tr><td>{{ index $alert.Labels "alertname" }}</td><td>{{ $alert.StartsAt }}</td></tr>
    {{ end }}
</table>
{{ end }}

啓動各個組件

啓動AlertManager

cd /home/admin/alertmanager-0.13.0.linux-amd64
./alertmanager --config.file=alert.yml

啓動Prometheus

cd /home/admin/prometheus-2.1.0.linux-amd64
./prometheus --config.file=prometheus.yml

啓動exporter

cd  /prometheus_exporters

./node_exporter & 

./redis_exporter redis//localhost:6379 & -web.listenaddress 0.0.0.0:9122

效果

收到的郵件