EFK教程(3) - ElasticSearch多實例部署

時間  2020-01-27
標籤 efk 教程 elasticsearch 實例 部署 欄目 日誌分析 简体版
原文   原文鏈接

EFK教程(3) - ElasticSearch多實例部署

基於ElasticSearch多實例架構，實現資源合理分配、冷熱數據分離node

做者：「發顛的小狼」，歡迎轉載與投稿linux

目錄

▪ 用途
▪ 架構
▪ 192.168.1.51 elasticsearch-data部署雙實例
▪ 192.168.1.52 elasticsearch-data部署雙實例
▪ 192.168.1.53 elasticsearch-data部署雙實例
▪ 測試nginx

用途

前情提要：json

▷ 在第一篇《EFK教程 - 快速入門指南》中，闡述了EFK的安裝部署，其中ES的架構爲三節點，即master、ingest、data角色同時部署在三臺服務器上。
▷ 在第二篇《EFK教程 - ElasticSearch高性能高可用架構》中，闡述了EFK的data/ingest/master角色的用途及分別部署三節點，在實現性能最大化的同時保障高可用服務器

前兩篇文章，ES集羣中只存在一個實例，而在本文中，將在一個集羣中部署多個ES實例，來實現資源合理分配。例如data服務器存在SSD與SAS硬盤，能夠將熱數據存放到SSD，而冷數據存放到SAS，實現數據冷熱分離。架構

在本文中，將爲data服務器建立2個實例，分別基於SSD和基於SAS硬盤，將nginx的9月份索引放在SAS盤上，其它的全放在SSD盤上app

架構

架構圖cors

EFK教程(3) - ElasticSearch多實例部署

服務器配置curl

EFK教程(3) - ElasticSearch多實例部署

192.168.1.51 elasticsearch-data部署雙實例

索引遷移（此步不能忽略）：將192.168.1.51上的索引放到其它2臺data節點上elasticsearch

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.52,192.168.1.53"
}'

確認當前索引存儲位置：確認全部索引不在192.168.1.51節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

EFK教程(3) - ElasticSearch多實例部署

停掉192.168.1.51的進程，修改目錄結構及配置：請自行按SSD和SAS硬盤掛載好數據盤

EFK教程(3) - ElasticSearch多實例部署

# 安裝包下載和部署請參考第一篇《EFK教程 - 快速入門指南》
cd /opt/software/
tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv /opt/elasticsearch /opt/elasticsearch-SAS
mv elasticsearch-7.3.2 /opt/
mv /opt/elasticsearch-7.3.2 /opt/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/elasticsearch-* -R
rm -rf /data/SAS/*
chown elasticsearch.elasticsearch /data/* -R
mkdir -p /opt/logs/elasticsearch-SAS
mkdir -p /opt/logs/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/logs/* -R

SAS實例/opt/elasticsearch-SAS/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.51-SAS
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch-SAS
network.host: 192.168.1.51

http.port: 9200
transport.port: 9300
# discovery.seed_hosts和cluster.initial_master_nodes 必定要帶上端口號，否則會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機只允行啓2個實例
node.max_local_storage_nodes: 2

SSD實例/opt/elasticsearch-SSD/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.51-SSD
path.data: /data/SSD
path.logs: /opt/logs/elasticsearch-SSD
network.host: 192.168.1.51

http.port: 9201
transport.port: 9301
# discovery.seed_hosts和cluster.initial_master_nodes 必定要帶上端口號，否則會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機只允行啓2個實例
node.max_local_storage_nodes: 2

SAS實例和SSD實例啓動方式

sudo -u elasticsearch /opt/elasticsearch-SAS/bin/elasticsearch
sudo -u elasticsearch /opt/elasticsearch-SSD/bin/elasticsearch

確認SAS和SSD已啓2實例

curl "http://192.168.1.31:9200/_cat/nodes?v"

EFK教程(3) - ElasticSearch多實例部署

192.168.1.52 elasticsearch-data部署雙實例

索引遷移（此步不能忽略）：將192.168.1.52上的索引放到其它2臺data節點上

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.51,192.168.1.53"
}'

確認當前索引存儲位置： 確認全部索引不在192.168.1.52節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

EFK教程(3) - ElasticSearch多實例部署

EFK教程(3) - ElasticSearch多實例部署

停掉192.168.1.52的進程，修改目錄結構及配置：請自行按SSD和SAS硬盤掛載好數據盤

EFK教程(3) - ElasticSearch多實例部署

# 安裝包下載和部署請參考第一篇《EFK教程 - 快速入門指南》
cd /opt/software/
tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv /opt/elasticsearch /opt/elasticsearch-SAS
mv elasticsearch-7.3.2 /opt/
mv /opt/elasticsearch-7.3.2 /opt/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/elasticsearch-* -R
rm -rf /data/SAS/*
chown elasticsearch.elasticsearch /data/* -R
mkdir -p /opt/logs/elasticsearch-SAS
mkdir -p /opt/logs/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/logs/* -R

SAS實例/opt/elasticsearch-SAS/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.52-SAS
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch-SAS
network.host: 192.168.1.52

http.port: 9200
transport.port: 9300
# discovery.seed_hosts和cluster.initial_master_nodes 必定要帶上端口號，否則會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機只允行啓2個實例
node.max_local_storage_nodes: 2

SSD實例/opt/elasticsearch-SSD/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.52-SSD
path.data: /data/SSD
path.logs: /opt/logs/elasticsearch-SSD
network.host: 192.168.1.52

http.port: 9201
transport.port: 9301
# discovery.seed_hosts和cluster.initial_master_nodes 必定要帶上端口號，否則會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機只允行啓2個實例
node.max_local_storage_nodes: 2

SAS實例和SSD實例啓動方式

sudo -u elasticsearch /opt/elasticsearch-SAS/bin/elasticsearch
sudo -u elasticsearch /opt/elasticsearch-SSD/bin/elasticsearch

確認SAS和SSD已啓2實例

curl "http://192.168.1.31:9200/_cat/nodes?v"

EFK教程(3) - ElasticSearch多實例部署

192.168.1.53 elasticsearch-data部署雙實例

索引遷移（此步不能忽略）：必定要作這步，將192.168.1.53上的索引放到其它2臺data節點上

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.51,192.168.1.52"
}'

確認當前索引存儲位置：確認全部索引不在192.168.1.52節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

EFK教程(3) - ElasticSearch多實例部署

EFK教程(3) - ElasticSearch多實例部署

停掉192.168.1.53的進程，修改目錄結構及配置：請自行按SSD和SAS硬盤掛載好數據盤

EFK教程(3) - ElasticSearch多實例部署

# 安裝包下載和部署請參考第一篇《EFK教程 - 快速入門指南》
cd /opt/software/
tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv /opt/elasticsearch /opt/elasticsearch-SAS
mv elasticsearch-7.3.2 /opt/
mv /opt/elasticsearch-7.3.2 /opt/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/elasticsearch-* -R
rm -rf /data/SAS/*
chown elasticsearch.elasticsearch /data/* -R
mkdir -p /opt/logs/elasticsearch-SAS
mkdir -p /opt/logs/elasticsearch-SSD
chown elasticsearch.elasticsearch /opt/logs/* -R

SAS實例/opt/elasticsearch-SAS/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.53-SAS
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch-SAS
network.host: 192.168.1.53

http.port: 9200
transport.port: 9300
# discovery.seed_hosts和cluster.initial_master_nodes 必定要帶上端口號，否則會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機只允行啓2個實例
node.max_local_storage_nodes: 2

SSD實例/opt/elasticsearch-SSD/config/elasticsearch.yml配置

cluster.name: my-application
node.name: 192.168.1.53-SSD
path.data: /data/SSD
path.logs: /opt/logs/elasticsearch-SSD
network.host: 192.168.1.53

http.port: 9201
transport.port: 9301
# discovery.seed_hosts和cluster.initial_master_nodes 必定要帶上端口號，否則會走http.port和transport.port端口
discovery.seed_hosts: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
cluster.initial_master_nodes: ["192.168.1.31:9300","192.168.1.32:9300","192.168.1.33:9300"]
http.cors.enabled: true
http.cors.allow-origin: "*"

node.master: false
node.ingest: false
node.data: true

# 本機只允行啓2個實例
node.max_local_storage_nodes: 2

SAS實例和SSD實例啓動方式

sudo -u elasticsearch /opt/elasticsearch-SAS/bin/elasticsearch
sudo -u elasticsearch /opt/elasticsearch-SSD/bin/elasticsearch

確認SAS和SSD已啓2實例

curl "http://192.168.1.31:9200/_cat/nodes?v"

EFK教程(3) - ElasticSearch多實例部署

測試

將全部索引移到SSD硬盤上

# 下面的參數會在後面的文章講解，此處照抄便可
curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._host_ip": "",
  "index.routing.allocation.include._host": "",
  "index.routing.allocation.include._name": "",
  "index.routing.allocation.include._ip": "",
  "index.routing.allocation.require._name": "*-SSD"
}'

確認全部索引全在SSD硬盤上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

EFK教程(3) - ElasticSearch多實例部署

將nginx9月份的日誌索引遷移到SAS硬盤上

curl -X PUT "192.168.1.31:9200/nginx_*_2019.09/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.require._name": "*-SAS"
}'

確認nginx9月份的日誌索引遷移到SAS硬盤上

curl "http://192.168.1.31:9200/_cat/shards"

EFK教程(3) - ElasticSearch多實例部署

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程