es按期刪除數據 es按期刪除數據

es按期刪除數據

一、按期刪除索引

使用sentinl報警後，會產生大量以下索引，雖然不佔空間，但時間久了也很差，故寫個腳本按期刪除

腳本以下：

1 #!/bin/bash
2 #只保留5天內的日誌索引
3 LAST_DATA=`date -d "-5 days" "+%Y.%m.%d"`
4 #刪除上個月份全部的索引
5 curl -XDELETE 'http://10.139.xx.xx:9200/*-'${LAST_DATA}''

再在設置一個定時策略便可

1 0 1 * * * /data1/elk/scripts/clear-index.sh

 

二、按期刪除索引

Curator 是elasticsearch 官方的一個索引管理工具，能夠刪除、建立、關閉、段合併等等功能

安裝

參考官網：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html

pip install elasticsearch-curator

安裝完若是curator 和curator_cli說明安裝成功

curator核心在於倆個配置文件，配置文件名稱隨意無要求：

配置文件config.yml：配置要鏈接的ES地址、日誌配置、日誌級別等；

執行文件action.yml: 配置要執行的操做(可批量）、配置索引的格式（前綴匹配、正則匹配方式等）

config.yml樣例

具體參數解析見官網：https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/configfile.html

client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False
 
logging:
  loglevel: INFO
  logfile: /var/log/elasticsearch-curator.log
  logformat: default
  blacklist: []

action.yml樣例(刪除3天前的數據)：

參數具體意思參見官網：https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/actionfile.html

actions:
  1:
    action: delete_indices
    description: >-
      Delete metric indices older than 3 days (based on index name), for
      zou_data-2018-05-01
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options :
      ignore_empty_list: True
      disable_action: True
    filters:
      - filtertype: pattern
             kind: regex
             value: '^(zou_data-).*$'
      - filtertype: age
             source: name
             direction: older
             timestring: '%Y-%m-%d'
             unit: days
             unit_count: 3

 

運行curator

單次運行

curator --config config.yml action.yml 

定時任務運行

0 0 */1 * * curator --config /opt/elasticsearch-curator/config.yml /opt/elasticsearch-curator/action.yml

 

 

三、按期刪除索引內的數據

 

#!/bin/bash

indexs=` curl -X GET 'http://10.10.10.10:9200/_cat/indices?v' | awk '{print $3}' | grep -vE '(kibana|index|watcher|monitoring)'`

for index in $indexs
  do
     curl  -X POST "10.139.34.129:9200/$index/_delete_by_query?pretty" -H 'Content-Type:application/json' -d '
      {
         "query": { 
              "bool": {
                   "must": [
                      {
                        "range": {
                           "@timestamp": {
                               "gte": "now-7d",
                               "lte": "now",
                               "format": "epoch_millis"
                              }
                         }
                     }
                   ],
                   "must_not": []
              }  
         }  
     }'
     echo "已清除$index 索引內七天前數據~"
   done

 

0 1 * * * /data1/elk/scripts/clear-data.sh

 

ES的刪除操做，不會當即生效，跟更新操做相似。只是會被標記爲已刪除狀態，ES後期會自動刪除。

 

 

es啓動腳本

#!/bin/bash
#set -x
cd `dirname $0`
data_dir=/data1/elk/elasticsearch

if [ ! -d $data_dir/data ]; then
    mkdir $data_dir/data && chown -R dev.dev $data_dir
fi

bin_dir=$data_dir/bin

PID=`ps -ef | grep elasticsearch | grep -v grep  | grep root | grep -v bash |awk '{print $2}'`

if [ -n "$PID" ]
        then kill -9 $PID
        echo "before: $PID"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/elasticsearch"  >> /dev/null 2>&1 &
        sleep 3
        P=`ps -ef | grep elasticsearch | grep -v grep |  grep root | grep -v bash |awk '{print $2}'`
        echo "now   : $P"
else 
        echo "starting"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/elasticsearch"  >> /dev/null 2>&1 &
        P=`ps -ef | grep elasticsearch | grep -v grep  |  grep root | grep -v bash |awk '{print $2}'`
        echo "now   : $P"
fi

 

kinaba啓動腳本

#!/bin/bash
#set -x
cd `dirname $0`
data_dir=/data1/elk/kibana

if [ ! -d $data_dir/data ]; then
    mkdir $data_dir/data && chown -R dev.dev $data_dir
fi

bin_dir=$data_dir/bin

PID=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`

if [ -n "$PID" ]
        then kill -9 $PID
        echo "before: $PID"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/kibana"  >> $data_dir/logs/kibana.log 2>&1 &
        sleep 3
        P=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`
        echo "now   : $P"
else 
        echo "starting"
        cd $bin_dir &&  nohup su - dev -c "$bin_dir/kibana"  >> $data_dir/logs/kibana.log 2>&1 &
        P=`netstat -nlpt | grep 5601 | awk '{print $7}' | cut -d / -f1`
        echo "now   : $P"
fi