Zabbix後端存儲ES的優化實踐

時間 2020-03-26

原文原文鏈接

場景分析

因爲公司zabbix的歷史數據存儲在elasticsearch中，有個需求是儘量地把監控的歷史數據存儲的長一點，最好是一年，目前的狀況是三臺ES節點，天天監控歷史數據量有5G，目前最多可存儲一個月的數據，超過30天的會被定時刪除，每臺內存分了8G，且所有使用機械硬盤，主分片爲5，副本分片爲1，查詢需求通常只獲取一週的歷史數據，偶爾會有查一個月到兩個月曆史數據的需求。php

節點規劃

爲了讓ES能存儲更長的歷史數據，我將節點數量增長至4節點，並將部分節點內存提升，部分節點採用SSD存儲html

192.168.179.133  200GSSD 4G內存  tag:hot node.name=es1
192.168.179.134  200GSSD 4G內存  tag:hot node.name=es2
192.168.179.135  500GHDD 64G內存  tag:cold node.name=es3
192.168.179.136  500GHDD 64G內存  tag:cold node.name=es4

優化思路

對數據mapping從新建模，對str類型的數據不進行分詞，採用冷熱節點對數據進行存儲，前七天數據的索引分片設計爲2主1副，索引存儲在熱節點上，超過七天的數據將被存儲在冷節點，並修改冷節點上的副本分片數爲0，ES提供了一個shrink的api來進行壓縮。因爲ES是基於Lucene的搜索引擎，Lucene的索引由多個segment組成，每個段都會消耗文件句柄，內存和CPU運行週期，段數量過多會使資源消耗變大，搜索也會變慢，這裏我將前一天的索引分片強制合併爲1個segment，修改refresh的時間間隔至60s，減小段的產生頻率。對超過3個月的索引進行關閉。以上操做均使用ES的管理工具curator來定時執行。node

zabbix與ES的對接操做

1.修改/etc/zabbix/zabbix_server.conf，添加以下內容

ES地址填寫集羣中任意一個節點就能夠web

HistoryStorageURL=192.168.179.133:9200
HistoryStorageTypes=str,text,log,uint,dbl
HistoryStorageDateIndex=1

2.修改/etc/zabbix/web/zabbix.conf.php，添加以下內容

global $DB, $HISTORY;
$HISTORY['url']   = 'http://192.168.179.133:9200';
// Value types stored in Elasticsearch.
$HISTORY['types'] = ['str', 'text', 'log','uint','dbl'];

3.修改ES配置文件，添加冷熱節點的標籤

vim elasticsearch.yml
熱節點配置vim

node.attr.box_type=hot

冷節點配置後端

node.attr.box_type=cold

3.在es上建立模板和管道

每種數據類型的模板都須要建立，能夠根據elasticsearch.map文件來獲取api的信息，模板定義內容有匹配的索引，主副分片數設置，refresh間隔，新建索引分配節點設置以及mapping的設置，這裏我只是以uint和str數據的索引爲例api

PUT _template/uint_template
{
   "template": "uint*",
   "index_patterns": ["uint*"],
   "settings" : {
      "index" : {
         "routing.allocation.require.box_type": "hot",
         "refresh_interval": "60s",
         "number_of_replicas" : 1,
         "number_of_shards" : 2
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "type" : "long"
            }
         }
      }
   }
}

PUT _template/str_template
{
   "template": "str*",
   "index_patterns": ["str*"],
   "settings" : {
      "index" : {
         "routing.allocation.require.box_type": "hot",
         "refresh_interval": "60s",
         "number_of_replicas" : 1,
         "number_of_shards" : 2
      }
   },
   "mappings" : {
      "values" : {
         "properties" : {
            "itemid" : {
               "type" : "long"
            },
            "clock" : {
               "format" : "epoch_second",
               "type" : "date"
            },
            "value" : {
               "index" : false,
               "type" : "keyword"
            }
         }
      }
   }
}

定義管道的做用是對寫入索引以前的數據進行預處理，使其按天產生索引。app

PUT _ingest/pipeline/uint-pipeline
{
  "description": "daily uint index naming",
  "processors": [
    {
      "date_index_name": {
        "field": "clock",
        "date_formats": ["UNIX"],
        "index_name_prefix": "uint-",
        "date_rounding": "d"
      }
    }
  ]
}
PUT _ingest/pipeline/str-pipeline
{
  "description": "daily str index naming",
  "processors": [
    {
      "date_index_name": {
        "field": "clock",
        "date_formats": ["UNIX"],
        "index_name_prefix": "str-",
        "date_rounding": "d"
      }
    }
  ]
}

4.修改完成後重啓zabbix,並查看zabbix是否有數據elasticsearch

systemctl restart zabbix-server

使用curator對索引進行操做

curator官方文檔地址以下
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/installation.htmlide

1.安裝curator

pip install -U elasticsearch-curator

2.建立curator配置文件

mkdir /root/.curator
vim /root/.curator/curator.yml
---
client:
  hosts:
    - 192.168.179.133
    - 192.168.179.134
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

3.編輯action.yml,定義action

將7天之前的索引分配到冷節點

1:
    action: allocation
    description: "Apply shard allocation filtering rules to the specified indices"
    options:
      key: box_type
      value: cold
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 7

將前一天的索引強制合併，每一個分片1個segment。

2:
    action: forcemerge
    description: "Perform a forceMerge on selected indices to 'max_num_segments' per shard"
    options:
      max_num_segments: 1
      delay:
      timeout_override: 21600 
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 1

修改冷節點得副本分片數量爲0

3:
    action: replicas
    description: "Set the number of replicas per shard for selected"
    options:
      count: 0
      wait_for_completion: True
      max_wait: 600
      wait_interval: 10
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 7

對超過六個月的索引進行關閉

4:
    action: close
    description: "Close selected indices"
    options:
      delete_aliases: false
      skip_flush: false
      ignore_sync_failures: false
    filters:
     - filtertype: pattern
       kind: regex
       value: '^(uint-|dbl-|str-).*$'
     - filtertype: age
       source: name
       direction: older
       timestring: '%Y-%m-%d'
       unit: days
       unit_count: 180

超過一年的索引進行刪除

5:
    action: delete_indices
    description: "Delete selected indices"
    options:
      continue_if_exception: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(uint-|dbl-|str-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'       
      unit: days
      unit_count: 365

4.執行curator進行測試

curator action.yml

5. 將curator操做寫進定時任務，天天執行

crontab -e
10 0 * * * curator /root/action.yml

以上就是對zabbix後端存儲elasticsearch存儲優化的所有實踐，參考連接
https://www.elastic.co/cn/blog/hot-warm-architecture-in-elasticsearch-5-x

歡迎關注我的公號「沒有故事的陳師傅」

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。