Elasticsearch,Filebeat,Kibana部署,添加圖表及elastalert報警

服務端安裝

Elasticsearch和Kibana（須要安裝openjdk1.8以上）

安裝方法：https://www.elastic.co
以Ubuntu爲例：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install elasticsearch
apt-get install kibana

elasticsearch配置

cat /etc/elasticsearch/elasticsearch.yml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.16.80.28
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.80.28","172.16.80.155"]

附件有文件elasticsearch.map

Kibana配置

cat /etc/kibana/kibana.yml

server.port: 5601
server.host: "172.16.80.28"
elasticsearch.url: "http://172.16.80.28:9200"

Elastalert（須要Python2.7）

下載 https://github.com/Yelp/elastalert/releases

進入elastalert目錄

pip install -r requirements.txt
python setup.py install
elastalert-create-index

前臺運行elastalert

python -m elastalert.elastalert --verbose --rule rules/my_rule.yaml

安裝參考:
https://elastalert.readthedocs.io/en/latest/running_elastalert.html#downloading-and-configuring
https://blog.csdn.net/df007df/article/details/54773391

elastalert配置

cat /etc/elastalert/rules/my_rule.yaml

es_host: 172.16.80.28
es_port: 9200
name: filebeat rules
type: frequency
index: filebeat*
num_events: 5
timeframe:
  hours: 1
filter:
- query:
    query_string:
      query: "message: *error*"
alert:
- "email"
email:
- "aaa@qq.com"
- "bbb@qq.com"
alert_text: "Ref Log http://172.16.80.28:5601/app/kibana"
smtp_host: smtp.exmail.qq.com
smtp_port: 25
smtp_ssl: false
smtp_auth_file: /etc/elastalert/rules/smtp_auth_file.yaml
from_addr:  aaa@qq.com

cat /etc/elastalert/rules/smtp_auth_file.yaml

user: "aaa@qq.com"
password: "郵箱密碼"

cat /etc/elastalert/config.yaml

rules_folder: /etc/elastalert/rules
run_every:
  minutes: 1
buffer_time:
  minutes: 15
es_host: 172.16.80.28
es_port: 9200
s_url_prefix: elasticsearch
writeback_index: elastalert_status
alert_time_limit:
  days: 2

elastalert配置註解

rules_folder：用來加載下一階段rule的設置，默認是example_rules
run_every：用來設置定時向elasticsearch發送請求
buffer_time：用來設置請求裏時間字段的範圍，默認是45分鐘
es_host：elasticsearch的host地址
es_port：elasticsearch 對應的端口號
use_ssl：可選的，選擇是否用SSL鏈接es，true或者false
es_username：es認證的username
es_password：es認證的password
writeback_index：elastalert產生的日誌在elasticsearch中的建立的索引
alert_time_limit：失敗重試的時間限制

告警參考
https://blog.csdn.net/gamer_gyt/article/details/52917116
https://elastalert.readthedocs.io/en/latest/ruletypes.html

客戶端安裝

Filebeat安裝

以Ubuntu爲例：

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install filebeat

filebeat配置

cat /etc/filebeat/filebeat.yml

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "172.16.80.28:5601"
output.elasticsearch:
  hosts: ["172.16.80.28:9200"]

EFK備註

日誌位置 /var/lib/elasticsearch/nodes/
日誌大小 暫時爲345Mb
日誌輪循 暫時不知道在哪設置

elasticsearch.map內容：

# uint mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "type" : "long"
            }
         }
      }
   }
}
 
# dbl mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "type" : "double"
            }
         }
      }
   }
}
 
# str mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "fields" : {
                  "analyzed" : {
                     "index" : true,
                     "type" : "text",
                     "analyzer" : "standard"
                  }
               },
               "index" : false,
               "type" : "text"
            }
         }
      }
   }
}
 
# text mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "fields" : {
                  "analyzed" : {
                     "index" : true,
                     "type" : "text",
                     "analyzer" : "standard"
                  }
               },
               "index" : false,
               "type" : "text"
            }
         }
      }
   }
}
 
# log mapping
{
   "settings" : {
      "index" : {
         "number_of_replicas" : 1,
         "number_of_shards" : 5
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "fields" : {
                  "analyzed" : {
                     "index" : true,
                     "type" : "text",
                     "analyzer" : "standard"
                  }
               },
               "index" : false,
               "type" : "text"
            }
         }
      }
   }
}

View Code