如何將Elasticsearch的快照備份至OSS

前言

Elasticsearch 是一個開源的分佈式 RESTful 搜索和分析引擎。它能夠在近實時條件下，存儲，查詢和分析海量的數據。它還支持將快照備份至HDFS/S3上面，而阿里雲OSS兼容S3的API，本文將介紹如何使用ES的Repository-S3插件將快照備份至OSS。

部署與配置

首先，咱們須要安裝repository-s3，能夠參考官方文檔：
https://www.elastic.co/guide/en/elasticsearch/plugins/7.2/repository-s3.html

啓動ES，咱們能夠從log中看到，ES已經load了這個plugin：

[2019-07-15T14:12:09,225][INFO ][o.e.p.PluginsService     ] [master] loaded module [aggs-matrix-stats]
[2019-07-15T14:12:09,225][INFO ][o.e.p.PluginsService     ] [master] loaded module [analysis-common]
[2019-07-15T14:12:09,225][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-common]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-geoip]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-user-agent]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-expression]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-mustache]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-painless]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [mapper-extras]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [parent-join]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [percolator]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [rank-eval]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded module [reindex]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded module [repository-url]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded module [transport-netty4]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [repository-s3]
[2019-07-15T14:12:12,375][INFO ][o.e.d.DiscoveryModule    ] [master] using discovery type [zen] and seed hosts providers [settings]
[2019-07-15T14:12:12,801][INFO ][o.e.n.Node               ] [master] initialized
[2019-07-15T14:12:12,802][INFO ][o.e.n.Node               ] [master] starting ...

而後，咱們須要將OSS使用的Access Key和Secret Key配置到ES去，分別執行下面的命令：

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

運行

首先，咱們建立一個備份：

[root@master ~]# curl -XPUT 'http://localhost:9200/_snapshot/test' -H 'Content-Type: application/json' -d '{ "type": "s3", "settings": { "bucket": "hadoop-oss-test", "endpoint": "oss-cn-zhangjiakou-internal.aliyuncs.com"} }'
{"acknowledged":true}

NOTE: 上面的命令默認使用https協議來傳輸數據，若是想使用http協議，須要將"protocol": "http", "disable_chunked_encoding": true加到settings裏面(這個特性將會在新版本發佈後可用)。

能夠使用下面的命令來確實建立是否成功：

[root@master ~]# curl -XGET localhost:9200/_snapshot/test?pretty
{
  "test" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "hadoop-oss-test",
      "endpoint" : "oss-cn-zhangjiakou-internal.aliyuncs.com"
    }
  }
}

咱們能夠寫入一些測試數據到ES，而後看下目前集羣的索引信息:

[root@master ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   sales    89ouBy6RQsuT34QRbn_jeQ  10   0     271786            0       15mb           15mb
green  open   customer fQCMEvXsQOu0UgMm1SAJlA   5   0      10000            0      717kb          717kb

假設咱們只備份sales索引：

[root@master ~]# curl -XPUT 'http://localhost:9200/_snapshot/test/sales' -H 'Content-Type: application/json' -d '{ "indices": "sales" }'
{"accepted":true}

而後咱們能夠從OSS控制檯看到備份的結果：

如今咱們再往sales索引裏面寫一些數據：

[root@master ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   sales    89ouBy6RQsuT34QRbn_jeQ  10   0     281502            0     15.6mb         15.6mb
green  open   customer fQCMEvXsQOu0UgMm1SAJlA   5   0      10000            0      717kb          717kb

咱們利用剛纔備份到OSS的快照來恢復sales索引，分別執行下面的命令：

[root@master ~]# curl -XPOST localhost:9200/sales/_close
{"acknowledged":true,"shards_acknowledged":true,"indices":{"sales":{"closed":true}}}
[root@master ~]# curl -XPOST 'http://localhost:9200/_snapshot/test/sales/_restore?pretty'
{
  "accepted" : true
}
[root@master ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   sales    89ouBy6RQsuT34QRbn_jeQ  10   0     271786            0       15mb           15mb
green  open   customer fQCMEvXsQOu0UgMm1SAJlA   5   0      10000            0      717kb          717kb

咱們能夠看到，sales索引跟以前的一致。

原文連接 本文爲雲棲社區原創內容，未經容許不得轉載。