Elastalert - ES告警組件

簡單可拓展，用於ES數據不一致，峯值等異常情形下的告警組件

工做方式

1. 週期性輪詢ES
2. 數據傳入elastalert規則引擎
3. 規則匹配則轉入elastalert告警器中

規則類型

• any：事件匹配指定filter
• change：指定字段在timeframe內發生值變更
• frequency：timeframe內發生幾回以上事件
• flatline：timeframe內發生幾回如下事件
• spike：事件頻率升高或下降
• blacklist/whitelist：指定字段出現黑白名單
• new_term：指定字段出現新值
• cardinality：指定字段去重基數超出或低於一個閾值

Kibana插件支持

• ./bin/kibana-plugin install 'https://git.bitsensor.io/front-end/elastalert-kibana-plugin/builds/artifacts/6.2.3/raw/artifact/elastalert-kibana-plugin-latest.zip?job=build'
• Kibana配置增長elastalert.serverHost: es主機

服務啓動

docker run -d -p 3030:3030 \
    -v `pwd`/config/elastalert.yaml:/opt/elastalert/config.yaml \
    -v `pwd`/config/config.json:/opt/elastalert-server/config/config.json \
    -v `pwd`/rules:/opt/elastalert/rules \
    -v `pwd`/rule_templates:/opt/elastalert/rule_templates \
    --net="host" \
    --name elastalert bitsensor/elastalert:latest

服務器配置示例

{
      "appName": "elastalert-server",
      "port": 3030,
      "elastalertPath": "/opt/elastalert",
      "verbose": false,
      "es_debug": false,
      "debug": false,
      "rulesPath": {
        "relative": true,
        "path": "/rules"
      },
      "templatesPath": {
        "relative": true,
        "path": "/rule_templates"
      }
    }

主配置示例

config.yaml

# This is the folder that contains the rule yaml files
    # Any .yaml file will be loaded as a rule
    rules_folder: /opt/elastalert/rules

    # In order to prevent error "rule duplicated"
    scan_subdirectories: false

    # How often ElastAlert will query Elasticsearch
    # The unit can be anything from weeks to seconds
    run_every:
      minutes: 1

    # ElastAlert will buffer results from the most recent
    # period of time, in case some log sources are not in real time
    buffer_time:
      minutes: 3

    # The Elasticsearch hostname for metadata writeback
    # Note that every rule can have its own Elasticsearch host
    es_host: elasticsearch-efk.$NAMESPACE.svc

    # The Elasticsearch port
    es_port: 9200

    # Optional URL prefix for Elasticsearch
    #es_url_prefix: elasticsearch

    # Connect with TLS to Elasticsearch
    use_ssl: False

    # Use SSL authentication with client certificates client_cert must be
    # a pem file containing both cert and key for client
    verify_certs: False
    #ca_certs: /path/to/cacert.pem
    #client_cert: /path/to/client_cert.pem
    #client_key: /path/to/client_key.key

    # GET request with body is the default option for Elasticsearch.
    # If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
    # See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
    # for details
    #es_send_get_body_as: GET

    # Option basic-auth username and password for Elasticsearch
    #es_username: someusername
    #es_password: somepassword

    # The index on es_host which is used for metadata storage
    # This can be a unmapped index, but it is recommended that you run
    # elastalert-create-index to set a mapping
    writeback_index: elastalert_status

    # If an alert fails for some reason, ElastAlert will retry
    # sending the alert until this time period has elapsed
    alert_time_limit:
      days: 2

    # Setup report smtp config
    from_addr: ElastAlert@yoursite.com
    smtp_host: smtp.exmail.qq.com
    smtp_port: 465
    smtp_ssl: True
    smtp_auth_file: /opt/elastalert/config/smtp_auth.yaml

smtp_auth.yaml

user: dev@maifusha.com
    password: yourpassword

告警規則示例

• 24h內數據進行規則測試：elastalert-test-rule [--config 可選配置路徑] rule.yaml
• filter type
  • query：支持模糊查詢
  • term：不支持模糊查詢，查詢時需改成 字段名.raw：查詢串

name: Report Web Error
    type: any
	#es_host: elasticsearch.example.com
	#es_port: 14900
    index: web.error-%Y.%m.%d
    use_strftime_index: True #只查詢匹配了index時間格式化正則的檢索
    #num_events: 3
    #timeframe:
    #  minutes: 3

	# ES的DSL查詢語法（最終查詢結構爲 {'filter': {'bool': {'must': [config.filter]}}}）
    filter:
      - query_string:
          query: "app.environment: (production staging)"

    query_delay:
      minutes: 1
    realert: #下一次alert的最小時間間隔，期間的alert被簡單丟棄(可與query_key結合使用)
      minutes: 1
    query_key: #alert去重的字段名（多個字段會致使檢索上新建一個組合字段名以用於查詢）
      - app.name
      - app.environment

    alert:
      - email
    email:
      - php@maifusha.com
    alert_subject: "[{1}報錯]{0}"
    alert_subject_args:
      - app.name
      - app.environment

	alert_text: "Something happened with {0} at {1}"
	alert_text_type: alert_text_only
	alert_text_args: ["username", "@timestamp"]
    include: ["ip_address", "hostname", "status"] #限制輸出的檢索字段

	# 聚合一段時間內的alert，並一次性發出
	aggregation:
	  schedule: '2 4 * * mon,fri'