（一）prometheus + grafana + alertmanager 配置主機監控

時間 2020-03-20

標籤 prometheus grafana alertmanager 配置主機監控简体版

原文原文鏈接

安裝請看http://www.javashuo.com/article/p-bfwlieic-do.html ，最好是對應的版本組件，不然可能會有差異。
html

（一）prometheus + grafana + alertmanager 配置主機監控node

（二）prometheus + grafana + alertmanager 配置Mysql監控mysql

（三）prometheus + grafana + alertmanager 配置Redis監控git

（四）prometheus + grafana + alertmanager 配置Kafka監控github

（五）prometheus + grafana + alertmanager 配置ES監控web

（一） prometheus + grafana + alertmanager配置主機監控sql

1. 配置prometheus（登錄到prometheus服務器，prometheus grafana alertmanager在同一臺服務器上） json

a. 打開vim /data/monitor/prometheus/conf/prometheus.yml文件。配置文件以下(全部的節點都是寫在json文件中)：小程序

global:vim

# Server端抓取數據的時間間隔

scrape_interval: 1m

# 評估報警規則的時間間隔

evaluation_interval: 1m

# 數據抓取的超時時間

scrape_timeout: 20s

# 加全局標籤

#external_labels:

#monitor: "usa"

# 鏈接alertmanager

alerting:

alertmanagers:

- static_configs:

- targets: ["localhost:9093"]

# 告警規則

rule_files:

- /data/monitor/prometheus/conf/rule/*.yml

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# 監控prometheus本機

- job_name: 'prometheus'

scrape_interval: 15s

static_configs:

- targets: ['10.8.9.2:9090']

# 監控指定主機

- job_name: 'node_resources'

scrape_interval: 1m

static_configs:

file_sd_configs:

- files:

- /data/monitor/prometheus/conf/node_conf/node_host_info.json

honor_labels: true

b. node主機json文件：

cat /data/monitor/prometheus/conf/node_conf/node_host_info.json

[

{

"labels": {

"desc": "ba_backend_10.8.9.35",

"group": "ba",

"host_ip": "10.8.9.35",

"hostname": "ba_backend"

"targets": [

"10.8.9.35:9100"

]

{

"labels": {

"desc": "ba3_10.8.32.67",

"group": "ba",

"host_ip": "10.8.32.67",

"hostname": "ba3"

"targets": [

"10.8.32.67:9100"

]

{

"labels": {

"desc": "ba1_10.8.46.117",

"group": "ba",

"host_ip": "10.8.46.117",

"hostname": "ba1"

"targets": [

"10.8.46.117:9100"

]

{

"labels": {

"desc": "ba2_10.8.80.126",

"group": "ba",

"host_ip": "10.8.80.126",

"hostname": "ba2"

"targets": [

"10.8.80.126:9100"

]

{

"labels": {

"desc": "openplatform_10.8.69.81",

"group": "openplatform",

"host_ip": "10.8.69.81",

"hostname": "openplatform"

"targets": [

"10.8.69.81:9100"

]

}

]

c. cd /data/monitor/prometheus, 而後 sh start.sh啓動prometheus，而後netstat -nltp |grep prometheus 查看9090端口是否已監聽

d. 在須要監控的服務器上（10.8.9.35，10.8.32.67，10.8.46.117，10.8.80.126，10.8.69.81）下載並安裝node_export（下載地址：https://pan.baidu.com/s/1gi-BM0rWWaGGKyWzUBFLPg），下載後解壓到/data/下，而後 cd /data/node_exporter , sh start.sh 啓動服務, netstat -nltp |grep node_exporter 查看9100是否已監聽

e. 登陸prometheus，在瀏覽器中打開 http://10.8.9.2:9090 ，先點菜單欄裏的Graph，而後在下面框中輸入 node_boot_time，最後點Execute查看下面是否有數據。

2. 配置grafana

a. /etc/init.d/grafana start 啓動grafana，而後 netstat -nltp |grep grafana 查看3000端口是否已監聽。

b. 在瀏覽器中打開 http://10.8.9.2:3000 登陸grafana，默認用戶名與密碼都是admin。

c. 首先點配置按鈕，而後在點Data Sources配置數據源。

d. 打開數據源頁面，而後點 Add data source

e. 打開選擇數據源頁面，選擇prometheus

f. 而後打開配置prometheus數據源頁面，在Name中輸入 Prometheus，並勾選Default, URL中填 http://localhost:9090（由於prometheus和grafana在同一臺機上），最後點Save&Test 進行保存。

g. 下載主機監控模板到本身電腦 https://pan.baidu.com/s/19RLrebLh5lI3nla4jaq1QA（固然你也能夠本身新建編輯，步驟是點+號，選擇圖表。）

h. 而後在grafana頁面中，點+，而後再點import

i. 而後在導入頁面，點Upload .json File,選擇剛下載下來的模板

j. 而後點change改變模板id，而後再點General, 而後點New Folder, 而後點Cancel，最後點Import，就會導入模板。

k. 而後就能夠看到數據展現了。

l. 固然你也可對現有的模板進行修改，或者新增或者設置等。

3. alertmanager配置

a. 配置規則，cat /data/monitor/prometheus/conf/rule/host.yml

groups:

- name: host_alert

rules:

### 硬盤 ###

# 默認系統盤告警策略

- alert: 主機系統盤80%

expr: floor(100-((node_filesystem_avail{device!="rootfs", mountpoint="/"}*100)/(node_filesystem_size{device!="rootfs", mountpoint="/"}*0.95))) >= 80

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

# 默認120G內數據盤告警策略

- alert: 主機數據盤90%

expr: (floor(100-((node_filesystem_avail{device!="rootfs", mountpoint="/data"}*100)/(node_filesystem_size{device!="rootfs", mountpoint="/data"}*0.95))) >= 90) and (node_filesystem_size{device!="rootfs", mountpoint="/data"}/1024/1024/1024 <= 120)

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

# 默認120G以上數據盤告警策略

- alert: 主機數據盤不足20G

expr: (floor(node_filesystem_avail{device!="rootfs", mountpoint="/data"}/1024/1024/1024) <= 20) and (node_filesystem_size{device!="rootfs", mountpoint="/data"}/1024/1024/1024 > 120)

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}G],告警初始時長爲3分鐘."

### CPU ###

# 默認CPU使用率告警策略

- alert: 主機CPU90%

expr: floor(100 - ( avg ( irate(node_cpu{mode='idle', hostname!~'consumer_service.*|backup_hk.*|bigdata.*master.*|3rdPart|htc_management|product_category_backend|sa_cluster_s.*'}[5m]) ) by (job, instance, hostname, desc) * 100 )) >= 90

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

# 持續時間較長的CPU使用率告警策略

- alert: 主機CPU90%

expr: floor(100 - ( avg ( irate(node_cpu{mode='idle', hostname=~'consumer_service.*|product_backend|sa_cluster_s.*'}[5m]) ) by (job, instance, hostname, desc) * 100 )) >= 90

for: 12m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲12分鐘."

# 持續時間較長的CPU使用率告警策略

- alert: 主機CPU90%

expr: floor(100 - ( avg ( irate(node_cpu{mode='idle', hostname=~'bigdata.*master.*|3rdPart|backup_hk.*'}[5m]) ) by (job, instance, hostname, desc) * 100 )) >= 90

for: 48m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲48分鐘."

### 內存 ###

# 默認內存使用率告警策略

- alert: 主機內存95%

expr: floor((node_memory_MemTotal - node_memory_MemFree - node_memory_Cached - node_memory_Buffers) / node_memory_MemTotal * 100) >= 95

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}%],告警初始時長爲3分鐘."

### 負載 ###

# 默認負載太高告警策略

- alert: 主機負載太高

expr: floor(node_load1{hostname!~"sa_cluster_s.*|bigdata.*master.*"}) >= 20

for: 3m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}],告警初始時長爲3分鐘."

# 持續時間較長的負載太高告警策略

- alert: 主機負載太高

expr: floor(node_load1{hostname=~"sa_cluster_s.*|bigdata.*master.*"}) >= 20

for: 12m

labels:

severity: warning

annotations:

description: "[{{ $labels.desc }}],告警值爲:[{{ $value }}],告警初始時長爲12分鐘."

b. 重啓prometheus，cd /data/monitor/prometheus , sh reload.sh

c. 配置alertmanager, cat /data/prometheus/alertmanager/conf/alertmanager.yml

global:

resolve_timeout: 2m

smtp_auth_password: q5AYahvxi3WLDap3 #發送郵箱密碼

smtp_auth_username: itliuqs@163.com #發送郵箱

smtp_from: itliuqs@163.com #發送郵箱

smtp_require_tls: false

smtp_smarthost: smtp.163.com:465 #發送服務器

wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/ #微信接口連接

inhibit_rules:

- equal:

- instance

source_match:

alertname: "主機CPU90%"

target_match:

alertname: "主機負載太高"

- equal:

- instance

source_match:

alertname: "mysql運行進程數5分鐘增加數>150"

target_match:

alertname: "mysql慢查詢5分鐘100條"

- equal:

- instance

source_match:

severity: error

target_match:

severity: warning

- equal:

- instance

source_match:

severity: fatal

target_match:

severity: error

- equal:

- service_name

source_match:

severity: error

target_match:

severity: warning

receivers:

- email_configs: #定義test發送人模塊

- html: '{{ template "email.default.html" . }}' #調用的模板

send_resolved: true

to: liuqs@126.com #將報警信息發給些郵箱，多人用|

name: test #發送人模板名

wechat_configs: #微信接收這些信息請看最下面的企業微信介紹

- agent_id: 1000002 #應用id

api_secret: hnyU1LTGnJUiBaCp47l3WVQLTEFF5RXyfNO751xlaHa #應用認證

corp_id: wwd397231fa801beaa #企業微信ID

send_resolved: true

to_user: LiuQingShan|liuqs #發送給企業微信通信人的Id 多我的就用|分開

- email_configs: #定義默認的發送人

- html: '{{ template "email.default.html" . }}'

send_resolved: true

to: liuqs@126.com

name: default_group

wechat_configs:

- agent_id: 1000002

api_secret: hnyU1LTGnJUiBaCp47l3WVQLTEFF5RXyfNO751xlaHa

corp_id: wwd397231fa801beaa

send_resolved: true

to_user: LiuQingShan

route: #定義資源報警規則

group_by:

- monitor

group_interval: 2m

group_wait: 30s

receiver: default_group

repeat_interval: 6h

routes:

- continue: true

match_re:

instance: 10.8.46.117:9100|10.8.80.126:9100|10.8.32.67:9100|10.8.9.35:9100|10.8.69.81:9100 #定義使用的資源

receiver: test #使用test發送人模板

templates:

- /data/monitor/alertmanager/template/*.tmpl #調用報警內容模板的路徑

d. 報警內容模板

cat/data/monitor/alertmanager/template/default.tmpl

{{ define "__alertmanager" }}AlertManager{{ end }}

{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}{{ end }}

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}Labels:

{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}

{{ end }}Annotations:

{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}

{{ end }}Source: {{ .GeneratorURL }}

{{ define "slack.default.fallback" }}{{ template "slack.default.title" . }} | {{ template "slack.default.titlelink" . }}{{ end }}

Alerts Firing:

Alerts Resolved:

<!--

#{{ template "__subject" . }}

#{{ .CommonAnnotations.SortedPairs.Values | join " " }}

Alerts Firing:

#{{ template "__text_alert_list" .Alerts.Firing }}

Labels:

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}

Start_time:{{ .StartsAt }}

告警主題: {{ .Annotations.summary }}

告警詳情: {{ .Annotations.description }}

Alerts Resolved:

#{{ template "__text_alert_list" .Alerts.Resolved }}

Labels:

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}

Start_time:{{ .StartsAt }}

End_time:{{ .EndsAt }}

告警主題: {{ .Annotations.summary }}

告警詳情: {{ .Annotations.description }}

#AlertmanagerUrl:

#{{ template "__alertmanagerURL" . }}

#{{- end }}

-->

Alerts Firing:

Alerts Resolved:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<!--

Style and HTML derived from https://github.com/mailgun/transactional-email-templates

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy

of this software and associated documentation files (the "Software"), to deal

in the Software without restriction, including without limitation the rights

to use, copy, modify, merge, publish, distribute, sublicense, and/or sell

copies of the Software, and to permit persons to whom the Software is

furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE

SOFTWARE.

-->

<title style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">{{ template "__subject" . }}</title>

</head>

{{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}

{{ .Name }}={{ .Value }}

</td>

</tr>

<a href="{{ template "__alertmanagerURL" . }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #FFF; text-decoration: none; line-height: 2em; font-weight: bold; text-align: center; cursor: pointer; display: inline-block; border-radius: 5px; text-transform: capitalize; background-color: #348eda; margin: 0; border-color: #348eda; border-style: solid; border-width: 10px 20px;">View in {{ template "__alertmanager" . }}</a>

</td>

</tr>

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Firing | len }}] Firing</strong>

</td>

</tr>

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Labels</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Start_time:</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ .StartsAt.Format "2006-01-02 15:04:05" }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ if gt (len .Annotations) 0 }}<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">Annotations</strong><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

{{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

<a href="{{ .GeneratorURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline; margin: 0;">Source</a><br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

</td>

</tr>

</td>

</tr>

<strong style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;">[{{ .Alerts.Resolved | len }}] Resolved</strong>

</td>

</tr>

{{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

{{ .StartsAt.Format "2006-01-02 15:04:05" }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ .EndsAt.Format "2006-01-02 15:04:05" }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />

{{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; margin: 0;" />{{ end }}

</td>

</tr>

</table>

</td>

</tr>

</table>

<td style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; vertical-align: top; text-align: center; color: #999; margin: 0; padding: 0 0 20px;" align="center" valign="top"><a href="{{ .ExternalURL }}" style="font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 12px; color: #999; text-decoration: underline; margin: 0;">Sent by {{ template "__alertmanager" . }}</a></td>

</tr>

</table>

</div></div>

</td>

</tr>

</table>

</body>

</html>

Alerts Firing:

Alerts Resolved:

cat /data/monitor/alertmanager/template/wechat.tmpl

{{ if gt (len .Alerts.Firing) 0 -}}告警: