因爲公司須要監控web攻擊行爲，而因某些緣由搭不了waf，纔不得不用ElastAlert進行告警，此爲前提。javascript
Elasticsearch 是一個分佈式、可擴展、實時的搜索與數據分析引擎。 它能從項目一開始就賦予你的數據以搜索、分析和探索的能力。php
Logstash是一款輕量級的日誌蒐集處理框架，能夠方便的把分散的、多樣化的日誌蒐集起來，並進行自定義的處理，而後傳輸到指定的位置， Kibana是一個開源的分析與可視化平臺，設計出來用於和Elasticsearch一塊兒使用的。你能夠用kibana搜索、查看、交互存放在Elasticsearch索引裏的數據，使用各類不一樣的圖表、表格、地圖等kibana可以很輕易地展現高級數據分析與可視化。html
ELK這一套軟件能夠看成一個MVC模型，logstash是controller層，Elasticsearch是一個model層，kibana是view層。首先將數據傳給logstash，它將數據進行過濾和格式化（轉成JSON格式），而後傳給Elasticsearch進行存儲、建搜索的索引，kibana提供前端的頁面再進行搜索和圖表可視化，它是調用Elasticsearch的接口返回的數據進行可視化。logstash和Elasticsearch是用Java寫的，kibana使用node.js框架。前端
安裝方法網上有好多，此處就再也不闡述。在試用了幾乎全部的安裝方法後，介紹下本人以爲比較快捷有效的安裝方法：java
elastalert目前還不支持elk6.0以上版本，本人就是由於版本問題而折騰了很久，因此在安裝elk的時候須要特別注意版本問題。node
個人服務器概況：
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial
在嘗試了衆多安裝方法後，仍是發如今官方網站下載deb包直接安裝最爲有效便捷。 （系統若爲centos，下載對應的rpm包）python
搜索版本下載nginx
Elasticsearch：5.5.2 Kibana：5.5.2 Logstash：6.0.0 filebeat：6.0.0 （輕量級的logstash，這個下載tar包）
理論上,Elasticsearch及Kibana版本爲5.x均可以，而Logstash與elastalert沒啥聯繫，因此Logstash（大於或等於Elasticsearch及Kibana的5.x版本）能向Elasticsearch傳遞日誌信息便可。git
下載完elk的deb包後，使用dpkg -i命令很快就能順利安裝。github
一樣，此處也只介紹本人的簡單配置。我這裏是只讓Logstash對外開放負責收集日誌，而Elasticsearch及Kibana僅在內網訪問，故Elasticsearch及Kibana並未開啓帳戶認證登錄，有須要開啓或其餘需求的讀者們請自行搜索。
/etc/elasticsearch/elasticsearch.yml:
# Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 127.0.0.1 # # Set a custom port for HTTP: # http.port: 9200
/etc/kibana/kibana.yml:
Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "localhost" # The URL of the Elasticsearch instance to use for all your queries. elasticsearch.url: "http://localhost:9200"
我這裏須要用到自定義的配置文件，故配置文件是自行建立的，放在/usr/share/logstash/bin中，取名爲filebeat_log.conf ：
input {
beats { port => 5044 client_inactivity_timeout => 90 codec => json } } filter { date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] target => ["datetime"] } geoip { source => "remote_addr" } mutate { remove_field => ["tags", "beat"] } } output { elasticsearch { hosts => "localhost:9200" index => "logstash-%{+YYYY.MM.dd}" } stdout { codec => rubydebug }
elasticsearch&kibana啓動
掃描新的單元&從新載入systemd：
systemctl daemon-reload
加入開機自啓動:
systemctl enable elasticsearch.service
systemctl enable kibana.service
啓動：
systemctl start elasticsearch
systemctl start kibana
查看狀態：
systemctl status elasticsearch
systemctl status kibana
logstash啓動
進入/usr/share/logstash/bin：
nohup ./logstash -f filebeat_log.conf> /dev/null 2>&1 &
一開始直接使用logstash進行日誌收集，發現資源消耗實在太大，無奈尋找解決方法，發現filebeat是一個輕量級的日誌傳輸工具，故使用filebeat做爲日誌收集，而logstash做爲中心過濾器向es傳遞日誌。
因此總體的架構如：
下面以tomcat爲例子，分享個人配置文件filebeat.yml（nginx的話，修改paths的路徑）：
filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - type: log # Change to true to enable this prospector configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /home/qy/apache-tomcat-9.0.1/logs/localhost_access_log.*.txt #- c:\programdata\elasticsearch\logs\* # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] document_type: tomcat-log scan_frequency: 15s ignore_older: 20m close_inactive: 12m clean_inactive: 30m close_removed: true clean_removed: true .... #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["188.88.88.88:5044"]
直接解壓下載的tar包，進入目錄修改配置文件。而後啓動filebeat：nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &
爲方便kibana分析和elastalert的取值，日誌的格式要爲json格式，上述的logstash配置文件已適配json格式。
公司的應用服務器中均爲nginx和tomcat，故本文只介紹tomcat及nginx的json格式配置方法，其餘服務器配置方法請自行搜索。
打開config/server.xml,在最後的位置修改log的輸出配置爲：
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="{"time":"%t","remote_addr":"%h","remote_user":"%l","request":"%r","status":"%s","body_bytes_sent":"%b","http_referer":"%{Referer}i","http_user_agent":"%{User-Agent}i","http_x_forwarded_for":" %{X-Forwarded-For}i","request_time":"%T","host":"%v","port":"%p"}"/>
而後重啓tomcat，即生效。
進入`/etc/nginx`打開`nginx.conf`，加入以下配置：
http {
##
# Basic Settings
##
sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; # server_tokens off; log_format logstash_json '{"time": "$time_local", ' '"remote_addr": "$remote_addr", ' '"remote_user": "$remote_user", ' '"request": "$request", ' '"status": "$status", ' '"body_bytes_sent": "$body_bytes_sent", ' '"http_referer": "$http_referer", ' '"http_user_agent": "$http_user_agent", ' '"http_x_forwarded_for": "$http_x_forwarded_for", ' '"request_time": "$request_time", ' '"request_length": "$request_length", ' '"host": "$http_host"}'; }
最後nginx -s reload便可
在通過上述的安裝及配置後，終於輪到咱們的主角—ElastAlert出來了，其餘的告警工具還有411 Alert Management、Elasticsearch watch ，請讀者們自行肯定須要使用哪一個。
ElastAlert使用python編寫，具備容易上手、文檔全等特色，雖然這個工具擁有如此多的優勢，在搭建過程仍是遇到了不少不少的未知錯誤，主要緣由是網上的資料大可能是針對es5.x之前的版本而沒什麼現成的資料可供參考。
git clone https://github.com/Yelp/elastalert.git cd elastalert python setup.py install //可能須要sudo Pip install -r requirements.txt //可能須要sudo cp config.yaml.example config.yaml
具體的功能本文就不一一介紹了，請自行前往官方文檔瞭解
安裝完成後會系統中會自帶三個命令：
elastalert-create-index、elastalert-rule-from-kibana 、elastalert-test-rule
使用elastalert-create-index，根據提示設置es後按回車默認便可。 配置完索引及配置文件後，可使用elastalert-test-rule進行測試。這裏有個bug，若是出現TransportError(400, u'search_phase_execution_exception', u'No mapping found for [alert_time] in order to sort on')之類的錯誤，在確認沒有其餘的問題時，能夠先刪除索引curl -XDELETE [http://localhost:9200/](http://localhost:9200/)*,再使用elastalert-create-index從新生成索引。
rules_folder: example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds run_every: seconds: 3 #每三秒向es請求數據 # ElastAlert will buffer results from the most recent # period of time, in case some log sources are not in real time buffer_time: minutes: 15 #日誌會延遲進入es，這裏是配置query的向前的時間範圍，這是15分鐘，即查詢 time[now-15m, now] # The Elasticsearch hostname for metadata writeback # Note that every rule can have its own Elasticsearch host es_host: 188.88.88.88 # The Elasticsearch port es_port: 9200 # Optional URL prefix for Elasticsearch #es_url_prefix: elasticsearch # Connect with TLS to Elasticsearch #use_ssl: True # Verify TLS certificates #verify_certs: True # GET request with body is the default option for Elasticsearch. # If it fails for some reason, you can pass 'GET', 'POST' or 'source'. # See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport # for details #es_send_get_body_as: GET # Option basic-auth username and password for Elasticsearch #es_username: someusername #es_password: somepassword # The index on es_host which is used for metadata storage # This can be a unmapped index, but it is recommended that you run # elastalert-create-index to set a mapping writeback_index: elastalert_status # If an alert fails for some reason, ElastAlert will retry # sending the alert until this time period has elapsed alert_time_limit: days: 1
以上各字段的解釋：
Rules_folder：用來加載下一階段rule的設置，默認是example_rules Run_every：用來設置定時向elasticsearch發送請求 Buffer_time：用來設置請求裏時間字段的範圍，默認是45分鐘 Es_host：elasticsearch的host地址 Es_port：elasticsearch 對應的端口號 Use_ssl：可選的，選擇是否用SSL鏈接es，true或者false Verify_certs：可選的，是否驗證TLS證書，設置爲true或者false，默認爲- true Es_username：es認證的username Es_password：es認證的password Es_url_prefix：可選的，es的url前綴（個人理解是https或者http） Es_send_get_body_as：可選的，查詢es的方式，默認的是GET Writeback_index：elastalert產生的日誌在elasticsearch中的建立的索引 Alert_time_limit：失敗重試的時間限制
在example_rules目錄中新建yaml配置文件 webattack_frequency.yaml,下面分開介紹這個配置文件的內容（下個小節將分享個人配置文件，此小節僅解釋其中的必要設置項）：
一、告警規則
ElastAlert支持11種告警規則，本文不一一介紹了，爲響應web攻擊行爲，本文選用的告警規則是frequency。
name: web attack
# (Required) # Type of alert. # the frequency rule type alerts when num_events events occur with timeframe time type: frequency # (Required, frequency specific) # Alert when this many documents matching the query occur within a timeframe num_events: 10 # (Required, frequency specific) # num_events must occur within this amount of time to trigger an alert timeframe: minutes: 1 # (Required) # Index to search, wildcard supported index: logstash-* #對應logstash的配置文件中output的elasticsearch index前綴 filter: - query_string: # sql insert xss detect query: "request: select.+(from|limit) OR request: union(.*?)select OR request: into.+(dump|out)file "
上述配置文件的意圖便是：在一分鐘內將匹配query裏面的sql注入規則，若匹配次數達到10次，即進行報警。
二、使用郵箱進行告警
ElastAlert提供了 10 多種通知的類型，本文選用的是郵箱告警，還有微信告警、釘釘告警，如有須要，請自行配置。
smtp_host: smtp.qiye.163.com smtp_port: 25 smtp_auth_file: /Users/qy/Downloads/work/elastalert/example_rules/smtp_auth_file.yaml #回覆給那個郵箱 email_reply_to: xxx@163.com #從哪一個郵箱發送 from_addr: xxx@163.com # (Required) # The alert is use when a match is found alert: - "email" # (required, email specific) # a list of email addresses to send alerts to email: - "shystartree@163.com" alert_subject: "web attack may be by {} at @{}" alert_subject_args: - remote_addr - time alert_text_type: alert_text_only alert_text: | 你好，服務器({})可能正在受到web攻擊，請採起手段阻止！！！！ ### 截止發郵件前匹配到的請求數：{} > 發生時間: {} > timestamp:{} > attacker's ip: {} > request: {} > status:{} > UA頭:{} >>> 參考來源：{} alert_text_args: - host - num_hits - time - "@timestamp" - remote_addr - request - status - http_user_agent - source
smtp_auth_file.yaml的配置內容會在下個小節給出，在這個配置中，我自定義了 alert 的內容，更爲精確地突出了攻擊者ip、受攻擊的服務器、攻擊事件等信息。
三、減小重複告警的頻率
在實際的使用中，若使用上述的配置，受到攻擊的時候郵箱將不斷地收到郵件，而這些郵件都對應着同一個攻擊實例，根本不必重複收取，因而，我使用了以下的配置：
# 用來區分報警，跟 realert 配合使用，在這裏意味着，
# 5 分鐘內若是有重複報警，那麼當 name 不一樣時，會當作不一樣的報警處理，能夠是數組
query_key: - name # 5 分鐘內相同的報警不會重複發送 realert: minutes: 5 # 指數級擴大 realert 時間，中間若是有報警， # 則按照 5 -> 10 -> 20 -> 40 -> 60 不斷增大報警時間到制定的最大時間， # 若是以後報警減小，則會慢慢恢復原始 realert 時間 exponential_realert: hours: 1
在本人實際測試的攻擊場景中，發現使用了exponential_realert後，會錯過不少告警（這些告警並非同一個攻擊實例），暫時不肯定緣由，還請讀者們自行肯定是否開啓該設置。
webattack_frequency.yaml及smtp_auth_file.yaml配置文件內容上述的4.4小節中對每一個配置都做了簡單的介紹，這裏就直接放出web攻擊預警的配置文件供各位讀者參考。
webattack_frequency.yaml:
# Alert when the rate of events exceeds a threshold
# (Optional) # Elasticsearch host #es_host: 188.88.88.88 # (Optional) # Elasticsearch port #es_port: 9200 # (OptionaL) Connect with SSL to Elasticsearch #use_ssl: True # (Optional) basic-auth username and password for Elasticsearch #es_username: someusername #es_password: somepassword # (Required) # Rule name, must be unique name: web attack realert: minutes: 5 # (Required) # Type of alert. # the frequency rule type alerts when num_events events occur with timeframe time type: frequency # (Required) # Index to search, wildcard supported index: logstash-* # (Required, frequency specific) # Alert when this many documents matching the query occur within a timeframe num_events: 10 # (Required, frequency specific) # num_events must occur within this amount of time to trigger an alert timeframe: #hours: 4 minutes: 1 # (Required) # A list of Elasticsearch filters used for find events # These filters are joined with AND and nested in a filtered query # For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html #filter: #- term: # some_field: "some_value" filter: - query_string: # sql insert xss detect query: "request: select.+(from|limit) OR request: union(.*?)select OR request: into.+(dump|out)file OR request: (base64_decode|sleep|benchmark|and.+1=1|and.+1=2|or%20|exec|information_schema|where%20|union%20|%2ctable_name%20|cmdshell|table_schema) OR request: (iframe|script|body|img|layer|div|meta|style|base|object|input|onmouseover|onerror|onload) OR request: .+etc.+passwd OR http_user_agent：(HTTrack|harvest|audit|dirbuster|pangolin|nmap|sqln|-scan|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|PycURL|zmeu|BabyKrokodil|netsparker|httperf|bench) OR status: (400|404|500|501) NOT (request:_health.html OR remote_addr:222.222.222.222 ) " smtp_host: smtp.qiye.163.com smtp_port: 25 smtp_auth_file: /Users/qy/Downloads/work/elastalert/example_rules/smtp_auth_file.yaml #回覆給那個郵箱 email_reply_to: xxx@163.com #從哪一個郵箱發送 from_addr: xxx@163.com # (Required) # The alert is use when a match is found alert: - "email" # (required, email specific) # a list of email addresses to send alerts to email: - "shystartree@163.com" alert_subject: "web attack may be by {} at @{}" alert_subject_args: - remote_addr - time alert_text_type: alert_text_only alert_text: | 你好，服務器