Elasticsearch日誌收集

Install pip if necessary

curl  "https://bootstrap.pypa.io/get-pip.py"  -o  "get-pip.py" python get-pip.py

Install Curator for Elasticsearch

Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by:

• Obtaining the full list of indices (or snapshots) from the cluster, as the actionable list
• Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list as needed.
• Perform various actions on the items which remain in the actionable list.

 

pip  install  elasticsearch-curator pip  install  click==6.7

 

Configure curator

mkdir  -p  /var/log/elastic touch  /var/log/elastic/curator .log mkdir  ~/.curator vi  ~/.curator /curator .yml

curator.yml

# Remember, leave a key empty if there is no value. None will be a string, ## not a Python "NoneType" client: hosts: [Elasticsearch Server IP] port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False   logging: loglevel: INFO logfile: /var/log/elastic/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3']

Have a test, now you can get the indices list
curator_cli show_indices

 

Create repository

Configure elasticseach.yml default in /etc/elasticsearch/elasticsearch.yml

elasticsearch.yml

path.repo:   /u01/elasticsearch/backup http.max_header_size: 16kb

Restart elasticsearch service (service elasticsearch restart) to make the configurations work.

Create repository elasticsearch. Ensure location points to a valid path which is configured in path.repo, accesable from all nodes.

curl -XPUT http: //localhost :9200 /_snapshot/es_backup  -H  "Content-Type: application/json"  -d @repository.json

repository.json

{     "type" :  "fs" ,     "settings" : {        "compress" :  true ,        "location" :  "/u01/elasticsearch/backup"     } }

 

Have a test

curl -XGET  'localhost:9200/_snapshot/_all?pretty=true'

Create curator yaml action files

daily_backup.yml

Customize the snapshot name in name option
action 1: backup all indices before today to repository elasticsearch with specified snapshot name
action 2: delete indices older than 185 days

daily_backup.yml

--- actions:    1:      action: snapshot      description: >-        Snapshot selected all indices to repository  'elasticsearch'  with the snapshot name      options:        repository: es_backup        name:  '<c4cert-{now/d-1d}>'        wait_for_completion: True        max_wait: 4800        wait_interval: 30      filters:      - filtertype: age        source : name        direction: older        unit: days        unit_count: 1        timestring:  "%Y.%m.%d"        2:      action: delete_indices      description: >-        Delete indices  which  is older than 185 days      filters:      - filtertype: age        source : name        direction: older        unit: days        unit_count: 185        timestring:  "%Y.%m.%d"

 

del_snapshot.yml
action 1: Delete snapshots from repository elasticsearch which is older than 185 days

del_snapshot.yml

---   actions:    1:      action: delete_snapshots      description: >-        Delete snapshots from repository  which  is older than 185 days      options:        repository: es_backup        retry_interval: 120        retry_count: 3      filters:      - filtertype: age        source : creation_date        direction: older        unit: days        unit_count: 185

 

restore.yml
action 1: Restore all indices in the most recent snapshot with state SUCCESS.

restore.yml

---   actions:    1:      action: restore      description: >-        Restore all indices  in  the most recent snapshot with state SUCCESS.  Wait        for  the restore to complete before continuing.  Do not skip the repository        filesystem access check.  Use the other options to define the index /shard        settings  for  the restore.      options:        repository: es_backup        # If name is blank, the most recent snapshot by age will be selected        name:        # If indices is blank, all indices in the snapshot will be restored        indices:        wait_for_completion: True        max_wait: 3600        wait_interval: 10      filters:      - filtertype: state        state: SUCCESS

 

Note: use --dry-run option to verify your action without any change. Find the dry run results in log path.
Curator --dry-run daily_backup.yml

 

Shell script and crontab

run.sh

#!/bin/sh curator  /u01/curator/del_snapshot .yml curator  /u01/curator/daily_backup .yml

crontab -e

Here configured the job run on every 3 AM

crontab

0 3 * * *  /bin/sh  /u01/curator/run .sh

 

Restore

Curator restore.yml

Tested OK in CERT env.

 

Some useful API 

# get all repositories curl -XGET  'localhost:9200/_snapshot/_all?pretty=true'   # delete repository curl -XDELETE  'localhost:9200/_snapshot/es-snapshot?pretty=true'   # show snapshots curator_cli show_snapshots --repository es_backup   # show indices curator_cli show_indices