基於prometheus + grafana + mysql + Telegram 監控告警
Prometheus是什麼
Prometheus(普羅米修斯)很是酷的開源監控系統。它支持多維度的指標數據模型，服務端經過HTTP協議定時拉取數據後，經過靈活的查詢語言，實現監控的目的。html
客戶端記錄相關指標數據，對外提供查詢接口。Prometheus服務端經過服務器發現機制找到客戶端，並定時抓取存儲爲時間序列數據。最後經過Grafana等圖表工具集成展現。java
Prometheus能夠作什麼
-
在業務層用做埋點系統
Prometheus支持各個主流開發語言（Go，java，python，ruby官方提供客戶端，其餘語言有第三方開源客戶端）。咱們能夠經過客戶端方面的對核心業務進行埋點。以下單流程、添加購物車流程。node -
在應用層用做應用監控系統
一些主流應用能夠經過官方或第三方的導出器，來對這些應用作核心指標的收集。如redis,mysql。python -
在系統層用做系統監控
除了經常使用軟件， prometheus也有相關係統層和網絡層exporter,用以監控服務器或網絡。mysql -
集成其餘的監控
prometheus還能夠經過各類exporte，集成其餘的監控系統，收集監控數據，如AWS CloudWatch,JMX，Pingdom等等。linux
開始安裝 prometheus
1 下載 nginx
https://prometheus.io/download/
2 解壓安裝git
tar xf prometheus-2.8.0.linux-amd64.tar.gz mv prometheus-2.8.0.linux-amd64 /usr/local/prometheus cd /usr/local/prometheus ./prometheus --version
3 啓動github
./prometheus
node節點安裝agent
1 下載web
https://prometheus.io/download/#node_exporter
2 安裝 agent
tar xf node_exporter-0.17.0.linux-amd64.tar.gz cd node_exporter-0.17.0.linux-amd64
3 啓動agent
./node_exporter
向prometheus 註冊節點
1 編輯配置文件
cd /usr/local/prometheus
# cat prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'linux-node' #添加一個job
static_configs:
- targets: ['10.10.25.151:9100'] 配置被監控的節點IP
labels:
instance: node1
服務端監聽 9090端口 prometheus web頁面 http://ServerIP:9090
客戶端監聽 9100端口
2 從新啓動 prometheus
./prometheus
3 訪問web頁面
http://10.10.25.110:9090
4 查看節點是否註冊
選擇 Status 裏面的 Target能夠看到咱們已經註冊的節點
5 簡單查看咱們的監控指標
之內存剩餘量爲例在輸入框輸入 node_memory_Cached_bytes
什麼是grafana
grafana 是一個開源的時序性統計和監控平臺，支持例如 elasticsearch、graphite、influxdb 等衆多的數據源，並以功能強大的界面編輯器著稱。
安裝grafana
1 下載
wget https://dl.grafana.com/oss/release/grafana-6.0.2-1.x86_64.rpm
2 安裝
yum install grafana-6.0.2-1.x86_64.rpm systemctl start grafana-server systemctl enable grafana-server grafana-server -v grafana-server 監聽端口爲 3000
3 訪問 grafana-server
http://ServerIP:3000 默認用戶名密碼爲： admin admin
4 添加數據源選擇 Prometheus
設置數據源的server地址和監聽端口並保存推出
5 添加Dashboard 也就是給節點添加可視化頁面
點擊‘ + ’ 選擇Import 輸入模板ID 8919
設置模板名稱，uid 數據源 保存便可看到監控頁面
注意 若是你是新的server會發現餅狀圖的數據加載不出，這個說明的是餅狀圖插件沒有安裝
6 安裝餅狀圖插件
grafana-cli plugins install grafana-piechart-panel systemctl restart grafana-server
安裝mysql做爲grafana 源
1 添加源
rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm yum repolist enabled | grep "mysql.*-community.*"
2 安裝 mysql-5.6
yum -y install mysql-community-server
3 啓動mysql並簡單安全設置
systemctl enable mysqld systemctl start mysqld systemctl status mysqld mysql_secure_installation 設置密碼一路Y
4 建立grafana 數據庫
create database grafana; create user grafana@'%' IDENTIFIED by 'grafana'; grant all on grafana.* to grafana@'%'; flush privileges;
5 修改配置文件鏈接mysql
vim /etc/grafana/grafana.ini [database] type = mysql host = 127.0.0.1:3306 name = grafana user = grafana password =grafana url = mysql://grafana:grafana@localhost:3306/grafana [session] provider = mysql provider_config = `grafana:grafana@tcp(127.0.0.1:3306)/grafana` systemctl restart grafana-server
6 重複grafana安裝配置便可
HAproxy指標導出器的安裝
1 進入opt目錄下載HAproxy節點指標導出器：
wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.9.0/haproxy_exporter-0.9.0.linux-amd64.tar.gz
2 解壓HAproxy節點指標導出器並刪除壓縮包：
tar -xvf haproxy_exporter-0.9.0.linux-amd64.tar.gz && rm haproxy_exporter-0.9.0.linux-amd64.tar.gz
3 修改文件夾名稱
mv haproxy_exporter-0.9.0.linux-amd64/ haproxy_exporter
4 追加HAproxy配置文件：
cat << EOF >> /etc/haproxy/haproxy.cfg
listen stats
mode http
bind 0.0.0.0:5000
stats enable
stats hide-version
stats uri /stats
stats realm Haproxy\ Statistics
stats auth admin:admin123
stats admin if TRUE #啓用管理功能
EOF
5 重啓HAproxy：
/etc/init.d/haproxy reload
6 進入對應文件夾後後臺運行導出器程序：
nohup ./haproxy_exporter --haproxy.scrape-uri="http://admin:admin123@localhost:5000/stats;csv" 2>&1 &
7 查看9100端口是否正常運行：
netstat -lntp
Windows節點指標導出器的安裝
1 下載Windows節點指標導出器：
https://github.com/martinlindhe/wmi_exporter/releases/download/v0.5.0/wmi_exporter-0.5.0-amd64.msi
2 打開Prowershell執行一下命令，注意程序所在路徑是否正確
msiexec /i C:\Users\Administrator\Downloads\wmi_exporter-0.5.0-amd64.msi ENABLED_COLLECTORS=iis,os,cpu,cs,memory,mssql,netframework_clrjit,netframework_clrinterop,netframework_clrexceptions,netframework_clrloading,netframework_clrlocksandthreads,netframework_clrmemory,net,service,system,tcp,textfile LISTEN_PORT=9182
配置grafana + Telegram 告警
1.對接Telegram官方接口@BotFather
2.輸入/newbot命令建立你本身的機器人
3.而後根據提示輸入機器人的名稱和用戶名，用戶名必須以bot結尾
4.這樣機器人就建立完成了，返回信息當中的token必須保存好備用
Done! Congratulations on your new bot. You will find it at t.me/zabbix999190411bot. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this. Use this token to access the HTTP API: 892821041:AAH9CPY5Ml-pybYgfd-cR-Vo1FZoual3cNE Keep your token secure and store it safely, it can be used by anyone to control your bot. For a description of the Bot API, see this page: https://core.telegram.org/bots/api
5.接下來使用API/bottoken/API方法getMe獲取本身的id
root@alex prometheus-2.8.1.linux-amd64]# curl https://api.telegram.org/bot892821041:AAH9CPY5Ml-pybYgfd-cR-Vo1FZoual3cNE/getMe
{"ok":true,"result":{"id":892821041,"is_bot":true,"first_name":"zabbix999190411bot","username":"zabbix999190411bot"}}
6.獲取羣ID
在Telegram新建group,而後添加成員剛建立的機器人，調用API方法getUPdates獲取羣ID
https://api.telegram.org/bot892821041:AAH9CPY5Ml-pybYgfd-cR-Vo1FZoual3cNE/getUpdates
{"ok":true,"result":[{"update_id":434572998,
"message":{"message_id":5,"from":{"id":787933444,"is_bot":false,"first_name":"Alex","last_name":"Liu","language_code":"zh-hans"},"chat":{"id":-234667255,
"title":"ZABBIX999","type":"group","all_members_are_administrators":true},"date":1554977987,"text":"/getMe","entities":[{"offset":0,"length":6,"type":"bot_command"}]}}]}
7 配置 grafana Alerting
配置完成後 點擊send test 測試
收到的測試告警
若是使用微信配置是相同的,只須要獲取企業帳號的相關ID便可
8 測試服務器報警
測試服務器告警以真實服務器爲例，測試內存數據告警
查看正式服務器內存
[root@lb-1 ~]# free -m
total used free shared buff/cache available
Mem: 1839 176 1382 8 281 1494
Swap: 2047 0 2047
添加魔板 8919 這裏的配置不作贅述以上面已經講過
接下來修改模板添加告警json,打開模板設置修改 json
修改json增長告警須要增長段json 塊,必需要注意的是配置wean告警必需要替換裏面的變量 $node 爲prometheus裏面配置文件相應的instance標籤的名字不然不能告警.
注意：下面的文件只作參考替換設置，版本不一樣的json文件並不適配！！！
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "使用 Node Exporter v0.16 0.17，精簡優化重要指標展現。包含：CPU 內存 磁盤 IO 網絡 溫度等監控指標。https://github.com/starsliao/Prometheus",
"editable": true,
"gnetId": 8919,
"graphTooltip": 0,
"id": 11,
"iteration": 1555301097807,
"links": [
{
"icon": "external link",
"tags": [],
"targetBlank": true,
"title": "更新node_exporter",
"tooltip": "",
"type": "link",
"url": "https://github.com/prometheus/node_exporter/releases"
},
{
"icon": "external link",
"tags": [],
"targetBlank": true,
"title": "更新儀表板",
"tooltip": "",
"type": "link",
"url": "https://grafana.com/dashboards/8919"
}
],
"panels": [
{
"content": "",
"editable": true,
"error": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 11,
"links": [],
"maxPerRow": 6,
"mode": "html",
"repeat": "node",
"repeatDirection": "h",
"scopedVars": {
"node": {
"selected": true,
"text": "nginxvlb",
"value": "nginxvlb"
}
},
"style": {},
"title": "nginxvlb",
"type": "text"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"datasource": "Prometheus",
"decimals": 1,
"description": "",
"format": "s",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 2,
"x": 0,
"y": 1
},
"hideTimeOverride": true,
"id": 15,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "time() - node_boot_time_seconds{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": false,
"instant": true,
"intervalFactor": 2,
"refId": "A",
"step": 40
}
],
"thresholds": "",
"title": "系統運行時間",
"type": "singlestat",
"valueFontSize": "100%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"datasource": "Prometheus",
"description": "",
"format": "short",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 2,
"x": 2,
"y": 1
},
"id": 14,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 6,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "count(count(node_cpu_seconds_total{instance=~\"nginxvlb\", mode='system'}) by (cpu))",
"format": "time_series",
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "A",
"step": 20
}
],
"thresholds": "",
"title": "CPU 核數",
"type": "singlestat",
"valueFontSize": "100%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "Prometheus",
"decimals": 2,
"description": "",
"format": "percent",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 3,
"x": 4,
"y": 1
},
"id": 167,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 12,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "100 - (avg(irate(node_cpu_seconds_total{instance=~\"nginxvlb\",mode=\"idle\"}[5m])) * 100)",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A",
"step": 20
}
],
"thresholds": "50,80",
"title": "CPU使用率（5m）",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "Prometheus",
"decimals": 2,
"description": "",
"format": "percent",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 3,
"x": 7,
"y": 1
},
"id": 20,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 12,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "avg(irate(node_cpu_seconds_total{instance=~\"nginxvlb\",mode=\"iowait\"}[5m])) * 100",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A",
"step": 20
}
],
"thresholds": "20,50",
"title": "CPU iowait（5m）",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "Prometheus",
"decimals": 0,
"description": "",
"format": "percent",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 3,
"x": 10,
"y": 1
},
"hideTimeOverride": false,
"id": 172,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 6,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "(1 - (node_memory_MemAvailable_bytes{instance=~\"nginxvlb\"} / (node_memory_MemTotal_bytes{instance=~\"nginxvlb\"})))* 100",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 20
}
],
"thresholds": "80,90",
"title": "內存使用率",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorPostfix": false,
"colorPrefix": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "Prometheus",
"decimals": 2,
"description": "",
"format": "short",
"gauge": {
"maxValue": 10000,
"minValue": null,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 3,
"x": 13,
"y": 1
},
"hideTimeOverride": false,
"id": 16,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 6,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "node_filefd_allocated{instance=~\"nginxvlb\"}",
"format": "time_series",
"instant": false,
"interval": "10s",
"intervalFactor": 1,
"refId": "B"
}
],
"thresholds": "7000,9000",
"title": "當前打開的文件描述符",
"type": "singlestat",
"valueFontSize": "70%",
"valueMaps": [],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "Prometheus",
"decimals": null,
"description": "",
"format": "percent",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 4,
"x": 16,
"y": 1
},
"id": 166,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 6,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"repeatDirection": "h",
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "100 - ((node_filesystem_avail_bytes{instance=~\"nginxvlb\",mountpoint=\"/\",fstype=~\"ext4|xfs\"} * 100) / node_filesystem_size_bytes {instance=~\"nginxvlb\",mountpoint=\"/\",fstype=~\"ext4|xfs\"})",
"format": "time_series",
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 20
}
],
"thresholds": "70,90",
"title": "根分區使用率",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"datasource": "Prometheus",
"decimals": null,
"description": "經過變量maxmount獲取最大的分區。",
"format": "percent",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 5,
"w": 4,
"x": 20,
"y": 1
},
"id": 154,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 6,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"repeat": null,
"repeatDirection": "h",
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "100 - ((node_filesystem_avail_bytes{instance=~\"nginxvlb\",mountpoint=\"$maxmount\",fstype=~\"ext4|xfs\"} * 100) / node_filesystem_size_bytes {instance=~\"nginxvlb\",mountpoint=\"$maxmount\",fstype=~\"ext4|xfs\"})",
"format": "time_series",
"interval": "10s",
"intervalFactor": 1,
"refId": "A",
"step": 20
}
],
"thresholds": "70,90",
"title": "最大分區($maxmount)使用率",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"datasource": "Prometheus",
"decimals": null,
"description": "",
"format": "bytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 2,
"y": 3
},
"id": 75,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"maxPerRow": 6,
"nullPointMode": "null",
"nullText": null,
"postfix": "",
"postfixFontSize": "70%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "node_memory_MemTotal_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"instant": true,
"intervalFactor": 1,
"legendFormat": "{{instance}}",
"refId": "A",
"step": 20
}
],
"thresholds": "",
"title": "內存總量",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"aliasColors": {
"15分鐘": "#6ED0E0",
"1分鐘": "#BF1B00",
"5分鐘": "#CCA300"
},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"grid": {},
"gridPos": {
"h": 6,
"w": 11,
"x": 0,
"y": 6
},
"height": "300",
"id": 13,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"maxPerRow": 6,
"nullPointMode": "null as zero",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"repeat": null,
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "node_load1{instance=~\"nginxvlb\"}",
"format": "time_series",
"instant": false,
"interval": "10s",
"intervalFactor": 2,
"legendFormat": "1m",
"metric": "",
"refId": "A",
"step": 20,
"target": ""
},
{
"expr": "node_load5{instance=~\"nginxvlb\"}",
"format": "time_series",
"instant": false,
"interval": "10s",
"intervalFactor": 2,
"legendFormat": "5m",
"refId": "B",
"step": 20
},
{
"expr": "node_load15{instance=~\"nginxvlb\"}",
"format": "time_series",
"instant": false,
"interval": "10s",
"intervalFactor": 2,
"legendFormat": "15m",
"refId": "C",
"step": 20
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "系統平均負載",
"tooltip": {
"msResolution": false,
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {
"/": "#eab839",
"/boot": "#bf1b00",
"/data": "#1f78c1"
},
"breakPoint": "100%",
"cacheTimeout": null,
"combine": {
"label": "Others",
"threshold": ""
},
"datasource": "Prometheus",
"decimals": 1,
"fontSize": "50%",
"format": "bytes",
"gridPos": {
"h": 6,
"w": 5,
"x": 11,
"y": 6
},
"hideTimeOverride": false,
"id": 171,
"interval": null,
"legend": {
"header": "",
"percentage": false,
"percentageDecimals": 0,
"show": true,
"sideWidth": 142,
"values": true
},
"legendType": "Right side",
"links": [],
"maxDataPoints": 3,
"nullPointMode": "connected",
"pieType": "pie",
"strokeWidth": "2",
"targets": [
{
"expr": "node_filesystem_size_bytes {instance=~\"nginxvlb\",fstype=~\"ext4|xfs\"}",
"format": "time_series",
"instant": true,
"interval": "10s",
"intervalFactor": 1,
"legendFormat": "{{mountpoint}}",
"refId": "A"
}
],
"title": "磁盤總空間",
"type": "grafana-piechart-panel",
"valueName": "current"
},
{
"columns": [],
"datasource": "Prometheus",
"fontSize": "120%",
"gridPos": {
"h": 6,
"w": 8,
"x": 16,
"y": 6
},
"id": 164,
"links": [],
"pageSize": null,
"scroll": true,
"showHeader": true,
"sort": {
"col": 11,
"desc": true
},
"styles": [
{
"alias": "Time",
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"pattern": "Time",
"type": "hidden"
},
{
"alias": "分區",
"colorMode": null,
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "mountpoint",
"thresholds": [
""
],
"type": "string",
"unit": "bytes"
},
{
"alias": "可用空間",
"colorMode": "value",
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "Value #A",
"thresholds": [
"10000000000",
"20000000000"
],
"type": "number",
"unit": "bytes"
},
{
"alias": "使用率",
"colorMode": "cell",
"colors": [
"rgba(50, 172, 45, 0.97)",
"rgba(237, 129, 40, 0.89)",
"rgba(245, 54, 54, 0.9)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "Value #B",
"thresholds": [
"70",
"90"
],
"type": "number",
"unit": "percentunit"
},
{
"alias": "總空間",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 1,
"link": false,
"mappingType": 1,
"pattern": "Value #C",
"thresholds": [],
"type": "number",
"unit": "bytes"
},
{
"alias": "文件系統",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"link": false,
"mappingType": 1,
"pattern": "fstype",
"thresholds": [],
"type": "number",
"unit": "short"
},
{
"alias": "",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"decimals": 2,
"pattern": "/.*/",
"preserveFormat": true,
"sanitize": false,
"thresholds": [],
"type": "hidden",
"unit": "short"
}
],
"targets": [
{
"expr": "node_filesystem_size_bytes{instance=~'nginxvlb',fstype=~\"ext4|xfs\"}",
"format": "table",
"hide": true,
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "C"
},
{
"expr": "node_filesystem_avail_bytes {instance=~'nginxvlb',fstype=~\"ext4|xfs\"}",
"format": "table",
"hide": false,
"instant": true,
"interval": "10s",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
},
{
"expr": "1-(node_filesystem_free_bytes{instance=~'nginxvlb',fstype=~\"ext4|xfs\"} / node_filesystem_size_bytes{instance=~'nginxvlb',fstype=~\"ext4|xfs\"})",
"format": "table",
"hide": false,
"instant": true,
"intervalFactor": 1,
"legendFormat": "",
"refId": "B"
},
{
"expr": "",
"format": "table",
"interval": "10s",
"intervalFactor": 1,
"legendFormat": "",
"refId": "D"
}
],
"title": "各分區可用空間",
"transform": "table",
"type": "table"
},
{
"aliasColors": {
"Idle - Waiting for something to happen": "#052B51",
"guest": "#9AC48A",
"idle": "#052B51",
"iowait": "#EAB839",
"irq": "#BF1B00",
"nice": "#C15C17",
"sdb_每秒I/O操做%": "#d683ce",
"softirq": "#E24D42",
"steal": "#FCE2DE",
"system": "#508642",
"user": "#5195CE",
"磁盤花費在I/O操做佔比": "#ba43a9"
},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 2,
"description": "node_disk_io_time_seconds_total：\n磁盤花費在輸入/輸出操做上的毫秒數。該值爲累加值。（Milliseconds Spent Doing I/Os）\n\nirate(node_disk_io_time_seconds_total[1m])：\n計算每秒的速率：(last值-last前一個值)/時間戳差值，即：1秒鐘內磁盤花費在I/O操做的時間佔比。",
"fill": 1,
"gridPos": {
"h": 7,
"w": 16,
"x": 0,
"y": 12
},
"id": 7,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"sideWidth": null,
"sort": null,
"sortDesc": null,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"maxPerRow": 6,
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"repeat": null,
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "avg(irate(node_cpu_seconds_total{instance=~\"nginxvlb\",mode=\"system\"}[1m]))",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "System",
"refId": "A",
"step": 20
},
{
"expr": "avg(irate(node_cpu_seconds_total{instance=~\"nginxvlb\",mode=\"user\"}[1m]))",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "User",
"refId": "B",
"step": 240
},
{
"expr": "avg(irate(node_cpu_seconds_total{instance=~\"nginxvlb\",mode=\"idle\"}[1m]))",
"format": "time_series",
"hide": true,
"intervalFactor": 2,
"legendFormat": "Idle",
"refId": "F",
"step": 240
},
{
"expr": "avg(irate(node_cpu_seconds_total{instance=~\"nginxvlb\",mode=\"iowait\"}[1m]))",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "Iowait",
"refId": "D",
"step": 240
},
{
"expr": "irate(node_disk_io_time_seconds_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{device}}_每秒I/O操做%",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "CPU使用率、磁盤每秒的I/O操做耗費時間（%）",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "percentunit",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"alert": { #這一段須要添加
"conditions": [
{
"evaluator": {
"params": [
1999999998
],
"type": "lt"
},
"operator": {
"type": "and"
},
"query": {
"params": [
"F",
"10s",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "alerting",
"for": "10s",
"frequency": "10s",
"handler": 1,
"message": "【Warning】[10.10.25.111]vlb服務器內存剩餘不足2G，請注意查看！",
"name": "[10.10.25.111]vlbserver mem is less then 2g",
"noDataState": "alerting",
"notifications": []
},
"aliasColors": {
"內存_Avaliable": "#6ED0E0",
"內存_Cached": "#EF843C",
"內存_Free": "#629E51",
"內存_Total": "#6d1f62",
"內存_Used": "#eab839",
"可用": "#9ac48a",
"總內存": "#bf1b00"
},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"decimals": 2,
"fill": 6,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 12
},
"height": "300",
"id": 156,
"legend": {
"alignAsTable": false,
"avg": false,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 3,
"links": [],
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "node_memory_MemTotal_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": false,
"instant": false,
"intervalFactor": 2,
"legendFormat": "總內存",
"refId": "A",
"step": 4
},
{
"expr": "node_memory_MemTotal_bytes{instance=~\"nginxvlb\"} - node_memory_MemAvailable_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "已用",
"refId": "B",
"step": 4
},
{
"expr": "node_memory_MemFree_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 2,
"legendFormat": "內存_Free",
"refId": "C",
"step": 4
},
{
"expr": "node_memory_Buffers_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 2,
"legendFormat": "內存_Buffers",
"refId": "D",
"step": 4
},
{
"expr": "node_memory_Cached_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 2,
"legendFormat": "內存_Cached",
"refId": "E",
"step": 4
},
{
"expr": "node_memory_MemAvailable_bytes{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "可用",
"refId": "F",
"step": 4
},
{
"expr": "node_memory_MemTotal_bytes{instance=~\"nginxvlb\"} - (node_memory_Cached_bytes{instance=~\"nginxvlb\"} + node_memory_Buffers_bytes{instance=~\"nginxvlb\"} + node_memory_MemFree_bytes{instance=~\"nginxvlb\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"refId": "G"
}
],
"thresholds": [
{###這一小段須要添加
"colorMode": "critical",
"fill": true,
"line": true,
"op": "lt",
"value": 1999999998
}
],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "內存信息",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {
"vda_write": "#6ED0E0"
},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"description": "Reads completed: 每一個磁盤分區每秒讀完成次數\n\nWrites completed: 每一個磁盤分區每秒寫完成次數\n\nIO now 每一個磁盤分區每秒正在處理的輸入/輸出請求數",
"fill": 2,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 19
},
"height": "300",
"id": 161,
"legend": {
"alignAsTable": false,
"avg": false,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_讀取$/",
"transform": "negative-Y"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "irate(node_disk_reads_completed_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{device}}_讀取",
"refId": "A",
"step": 10
},
{
"expr": "irate(node_disk_writes_completed_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "{{device}}_寫入",
"refId": "B",
"step": 10
},
{
"expr": "node_disk_io_now{instance=~\"nginxvlb\"}",
"format": "time_series",
"hide": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{device}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "磁盤讀寫速率（IOPS）",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "iops",
"label": "讀取（-）/寫入（+）I/O ops/sec",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {
"vda_write": "#6ED0E0"
},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"description": "Read bytes 每一個磁盤分區每秒讀取的比特數\nWritten bytes 每一個磁盤分區每秒寫入的比特數",
"fill": 2,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 19
},
"height": "300",
"id": 168,
"legend": {
"alignAsTable": false,
"avg": false,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_讀取$/",
"transform": "negative-Y"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "irate(node_disk_read_bytes_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{device}}_讀取",
"refId": "A",
"step": 10
},
{
"expr": "irate(node_disk_written_bytes_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 2,
"legendFormat": "{{device}}_寫入",
"refId": "B",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "磁盤讀寫容量大小",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "Bps",
"label": "讀取（-）/寫入（+）",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {
"vda": "#6ED0E0"
},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"description": "Read time ms 每一個磁盤分區讀操做花費的秒數\n\nWrite time ms 每一個磁盤分區寫操做花費的秒數\n\nIO time ms 每一個磁盤分區輸入/輸出操做花費的秒數\n\nIO time weighted 每一個磁盤分區輸入/輸出操做花費的加權秒數",
"fill": 3,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 19
},
"height": "300",
"id": 160,
"legend": {
"alignAsTable": false,
"avg": false,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/,*_讀取$/",
"transform": "negative-Y"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "irate(node_disk_io_time_seconds_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": true,
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{device}}",
"refId": "A",
"step": 10
},
{
"expr": "irate(node_disk_io_time_weighted_seconds_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "{{device}}_加權",
"refId": "D"
},
{
"expr": "irate(node_disk_read_time_seconds_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": false,
"interval": "",
"intervalFactor": 1,
"legendFormat": "{{device}}_讀取",
"refId": "B"
},
{
"expr": "irate(node_disk_write_time_seconds_total{instance=~\"nginxvlb\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{device}}_寫入",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "磁盤IO讀寫時間",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "s",
"label": "讀取（-）/寫入（+）",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 27
},
"height": "300",
"id": 157,
"legend": {
"alignAsTable": false,
"avg": false,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_out上傳$/",
"transform": "negative-Y"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "irate(node_network_receive_bytes_total{instance=~'nginxvlb',device!~'tap.*'}[5m])*8",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{device}}_in下載",
"refId": "A",
"step": 4
},
{
"expr": "irate(node_network_transmit_bytes_total{instance=~'nginxvlb',device!~'tap.*'}[5m])*8",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{device}}_out上傳",
"refId": "B",
"step": 4
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "網絡流量",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": "上傳（-）/下載（+）",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {
"TCP": "#6ED0E0"
},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"description": "CurrEstab - 當前狀態爲 ESTABLISHED 或 CLOSE-WAIT 的 TCP 鏈接數\n\nActiveOpens - 已從 CLOSED 狀態直接轉換到 SYN-SENT 狀態的 TCP 平均鏈接數(1分鐘內)\n\nPassiveOpens - 已從 LISTEN 狀態直接轉換到 SYN-RCVD 狀態的 TCP 平均鏈接數(1分鐘內)\n\nTCP_alloc - 已分配（已創建、已申請到sk_buff）的TCP套接字數量\n\nTCP_inuse - 正在使用（正在偵聽）的TCP套接字數量\n\nTCP_tw - 等待關閉的TCP鏈接數",
"fill": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 27
},
"height": "300",
"id": 158,
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"paceLength": 10,
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "node_netstat_Tcp_CurrEstab{instance=~'nginxvlb'}",
"format": "time_series",
"hide": false,
"interval": "10s",
"intervalFactor": 1,
"legendFormat": "ESTABLISHED",
"refId": "A",
"step": 20
},
{
"expr": "node_sockstat_TCP_tw{instance=~'nginxvlb'}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "TCP_tw",
"refId": "D"
},
{
"expr": "irate(node_netstat_Tcp_ActiveOpens{instance=~'nginxvlb'}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "ActiveOpens",
"refId": "B"
},
{
"expr": "irate(node_netstat_Tcp_PassiveOpens{instance=~'nginxvlb'}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "PassiveOpens",
"refId": "C"
},
{
"expr": "node_sockstat_TCP_alloc{instance=~'nginxvlb'}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "TCP_alloc",
"refId": "E"
},
{
"expr": "node_sockstat_TCP_inuse{instance=~'nginxvlb'}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "TCP_inuse",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "TCP 鏈接狀況",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 0,
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 35
},
"id": 169,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": true,
"hideZero": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null as zero",
"paceLength": 10,
"percentage": false,
"pointradius": 0.5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "node_hwmon_temp_celsius{instance=\"nginxvlb\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{chip}} {{sensor}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "硬件溫度",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "celsius",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": false,
"schemaVersion": 18,
"style": "dark",
"tags": [
"StarsL",
"Prometheus"
],
"templating": {
"list": [
{
"auto": true,
"auto_count": 30,
"auto_min": "10s",
"current": {
"text": "auto",
"value": "$__auto_interval_interval"
},
"hide": 0,
"label": "interval",
"name": "interval",
"options": [
{
"selected": true,
"text": "auto",
"value": "$__auto_interval_interval"
},
{
"selected": false,
"text": "1m",
"value": "1m"
},
{
"selected": false,
"text": "10m",
"value": "10m"
},
{
"selected": false,
"text": "30m",
"value": "30m"
},
{
"selected": false,
"text": "1h",
"value": "1h"
},
{
"selected": false,
"text": "6h",
"value": "6h"
},
{
"selected": false,
"text": "12h",
"value": "12h"
},
{
"selected": false,
"text": "1d",
"value": "1d"
},
{
"selected": false,
"text": "7d",
"value": "7d"
},
{
"selected": false,
"text": "14d",
"value": "14d"
},
{
"selected": false,
"text": "30d",
"value": "30d"
}
],
"query": "1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",
"refresh": 2,
"skipUrlSync": false,
"type": "interval"
},
{
"allFormat": "glob",
"allValue": null,
"current": {
"isNone": true,
"text": "None",
"value": ""
},
"datasource": "Prometheus",
"definition": "",
"hide": 0,
"includeAll": false,
"label": "環境",
"multi": false,
"multiFormat": "regex values",
"name": "env",
"options": [],
"query": "label_values(node_exporter_build_info,env)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allFormat": "glob",
"allValue": null,
"current": {
"isNone": true,
"text": "None",
"value": ""
},
"datasource": "Prometheus",
"definition": "",
"hide": 0,
"includeAll": false,
"label": "主機名",
"multi": false,
"multiFormat": "regex values",
"name": "name",
"options": [],
"query": "label_values(node_exporter_build_info{env='$env'},name)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allFormat": "glob",
"allValue": null,
"current": {
"tags": [],
"text": "nginxvlb",
"value": "nginxvlb"
},
"datasource": "Prometheus",
"definition": "",
"hide": 0,
"includeAll": false,
"label": "節點",
"multi": false,
"multiFormat": "regex values",
"name": "node",
"options": [],
"query": "label_values(node_exporter_build_info{name='$name'},instance)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"text": "/",
"value": "/"
},
"datasource": "Prometheus",
"definition": "",
"hide": 2,
"includeAll": false,
"label": "",
"multi": false,
"name": "maxmount",
"options": [],
"query": "query_result(topk(1,sort_desc (max(node_filesystem_size_bytes{instance=~'nginxvlb',fstype=~\"ext4|xfs\"}) by (mountpoint))))",
"refresh": 1,
"regex": "/.*\\\"(.*)\\\".*/",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-5m",
"to": "now"
},
"timepicker": {
"now": true,
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "browser",
"title": "nginxvlb",
"uid": "SW8eIbgZz",
"version": 4
}
須要重啓prometheus + grafana-server 服務
查看 Telegram羣組告警
若是須要其餘報警請本身編輯json文件.
相關文章
- 1. 基於Prometheus和Grafana的監控平臺 - 運維告警
- 2. 基於Prometheus+Grafana+Alertmanager監控Pulsar發釘釘告警
- 3. Prometheus+Grafana監控MySQL
- 4. prometheus+grafana監控mysql
- 5. Grafana+Prometheus 監控 MySQL
- 6. 基於Prometheus和Grafana的監控平臺 - MySQL監控
- 7. Grafana = 可視化分析 + 監控告警
- 8. Grafana+Prometheus監控mysql性能
- 9. Prometheus監控MySQL，Grafana展示
- 10. Grafana+Prometheus系統監控MySql
- 更多相關文章...
- • Spring基於Annotation裝配Bean - Spring教程
- • Spring使用AspectJ開發AOP：基於XML和基於Annotation - Spring教程
- • ☆基於Java Instrument的Agent實現
- • 漫談MySQL的鎖機制
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. 基於Prometheus和Grafana的監控平臺 - 運維告警
- 2. 基於Prometheus+Grafana+Alertmanager監控Pulsar發釘釘告警
- 3. Prometheus+Grafana監控MySQL
- 4. prometheus+grafana監控mysql
- 5. Grafana+Prometheus 監控 MySQL
- 6. 基於Prometheus和Grafana的監控平臺 - MySQL監控
- 7. Grafana = 可視化分析 + 監控告警
- 8. Grafana+Prometheus監控mysql性能
- 9. Prometheus監控MySQL，Grafana展示
- 10. Grafana+Prometheus系統監控MySql