EFK教程 - ElasticSearch角色分離

時間 2020-01-27

原文原文鏈接

經過將elasticsearch的data、ingest、master角色進行分離，搭建起高性能+高可用的ES架構node

做者：「發顛的小狼」，歡迎轉載與投稿linux

用途

在第一篇《EFK教程 - 快速入門指南》中，闡述了EFK的安裝部署，其中ES的架構爲三節點，即master、ingest、data角色同時部署在三臺服務器上。服務器

在本文中，將進行角色分離部署，而且每一個角色分別部署三節點，在實現性能最大化的同時保障高可用。架構

▷ elasticsearch的master節點：用於調度，採用普通性能服務器來部署
▷ elasticsearch的ingest節點：用於數據預處理，採用性能好的服務器來部署
▷ elasticsearch的data節點：用於數據落地存儲，採用存儲性能好的服務器來部署app

若不知道去哪找《EFK教程 - 快速入門指南》，可在主流搜索引擎裏搜索：
小慢哥 EFK教程 快速入門指南
或者
小慢哥 EFK教程 基於多節點ES的EFK安裝部署配置

架構

服務器配置cors

注意：此處的架構是以前的文章《EFK教程 - 快速入門指南》的拓展，所以請先按照《EFK教程 - 快速入門指南》完成部署curl

步驟說明

1️⃣ 部署3臺data節點，加入原集羣
2️⃣ 部署3臺ingest節點，加入原集羣
3️⃣ 將原有的es索引遷移到data節點
4️⃣ 將原有的es節點改形成master節點elasticsearch

elasticsearch-data部署

以前已完成了基礎的elasticsearch架構，現須要新增三臺存儲節點加入集羣，同時關閉master和ingest功能ide

elasticsearch-data安裝：3臺均執行相同的安裝步驟

tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv elasticsearch-7.3.2 /opt/elasticsearch
useradd elasticsearch -d /opt/elasticsearch -s /sbin/nologin
mkdir -p /opt/logs/elasticsearch
chown elasticsearch.elasticsearch /opt/elasticsearch -R
chown elasticsearch.elasticsearch /opt/logs/elasticsearch -R
# 數據盤須要elasticsearch寫權限
chown elasticsearch.elasticsearch /data/SAS -R

# 限制一個進程能夠擁有的VMA(虛擬內存區域)的數量要超過262144，否則elasticsearch會報max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
sysctl -p

elasticsearch-data配置

▷ 192.168.1.51 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.51
# 數據盤位置，若是有多個硬盤位置，用","隔開
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.51

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 關閉master功能
node.master: false
# 關閉ingest功能
node.ingest: false
# 開啓data功能
node.data: true

▷ 192.168.1.52 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.52
# 數據盤位置，若是有多個硬盤位置，用","隔開
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.52

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 關閉master功能
node.master: false
# 關閉ingest功能
node.ingest: false
# 開啓data功能
node.data: true

▷ 192.168.1.53 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.53
# 數據盤位置，若是有多個硬盤位置，用","隔開
path.data: /data/SAS
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.53

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 關閉master功能
node.master: false
# 關閉ingest功能
node.ingest: false
# 開啓data功能
node.data: true

elasticsearch-data啓動

sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch

elasticsearch集羣狀態

curl "http://192.168.1.31:9200/_cat/health?v"

elasticsearch-data狀態

curl "http://192.168.1.31:9200/_cat/nodes?v"

elasticsearch-data參數說明

status: green  # 集羣健康狀態
node.total: 6  # 有6臺機子組成集羣
node.data: 6  # 有6個節點的存儲
node.role: d  # 只擁有data角色
node.role: i  # 只擁有ingest角色
node.role: m  # 只擁有master角色
node.role: mid  # 擁master、ingest、data角色

elasticsearch-ingest部署

現須要新增三臺ingest節點加入集羣，同時關閉master和data功能

elasticsearch-ingest安裝：3臺es均執行相同的安裝步驟

tar -zxvf elasticsearch-7.3.2-linux-x86_64.tar.gz
mv elasticsearch-7.3.2 /opt/elasticsearch
useradd elasticsearch -d /opt/elasticsearch -s /sbin/nologin
mkdir -p /opt/logs/elasticsearch
chown elasticsearch.elasticsearch /opt/elasticsearch -R
chown elasticsearch.elasticsearch /opt/logs/elasticsearch -R

# 限制一個進程能夠擁有的VMA(虛擬內存區域)的數量要超過262144，否則elasticsearch會報max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]
echo "vm.max_map_count = 655350" >> /etc/sysctl.conf
sysctl -p

elasticsearch-ingest配置

▷ 192.168.1.41 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.41
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.41

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 關閉master功能
node.master: false
# 開啓ingest功能
node.ingest: true
# 關閉data功能
node.data: false

▷ 192.168.1.42 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.42
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.42

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 關閉master功能
node.master: false
# 開啓ingest功能
node.ingest: true
# 關閉data功能
node.data: false

▷ 192.168.1.43 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.43
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.43

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 關閉master功能
node.master: false
# 開啓ingest功能
node.ingest: true
# 關閉data功能
node.data: false

elasticsearch-ingest啓動

sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch

elasticsearch集羣狀態

curl "http://192.168.1.31:9200/_cat/health?v"

elasticsearch-ingest狀態

curl "http://192.168.1.31:9200/_cat/nodes?v"

elasticsearch-ingest參數說明

status: green  # 集羣健康狀態
node.total: 9  # 有9臺機子組成集羣
node.data: 6  # 有6個節點的存儲
node.role: d  # 只擁有data角色
node.role: i  # 只擁有ingest角色
node.role: m  # 只擁有master角色
node.role: mid  # 擁master、ingest、data角色

elasticsearch-master部署

首先，將上一篇《EFK教程 - 快速入門指南》中部署的3臺es（192.168.1.3一、192.168.1.3二、192.168.1.33）改爲只有master的功能，所以須要先將這3臺上的索引數據遷移到本次所作的data節點中

1️⃣ 索引遷移：必定要作這步，將以前的索引放到data節點上

curl -X PUT "192.168.1.31:9200/*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._ip": "192.168.1.51,192.168.1.52,192.168.1.53"
}'

2️⃣ 確認當前索引存儲位置：確認全部索引不在192.168.1.3一、192.168.1.3二、192.168.1.33節點上

curl "http://192.168.1.31:9200/_cat/shards?h=n"

elasticsearch-master配置

注意事項：修改配置，重啓進程，須要一臺一臺執行，要確保第一臺成功後，再執行下一臺。重啓進程的方法：因爲上一篇文章《EFK教程 - 快速入門指南》裏，是執行命令跑在前臺，所以直接ctrl - c退出再啓動便可，啓動命令以下

sudo -u elasticsearch /opt/elasticsearch/bin/elasticsearch

▷ 192.168.1.31 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.31
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.31

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#開啓master功能
node.master: true
#關閉ingest功能
node.ingest: false
#關閉data功能
node.data: false

▷ 192.168.1.32 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.32
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.32

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#開啓master功能
node.master: true
#關閉ingest功能
node.ingest: false
#關閉data功能
node.data: false

▷ 192.168.1.33 /opt/elasticsearch/config/elasticsearch.yml

cluster.name: my-application
node.name: 192.168.1.33
path.logs: /opt/logs/elasticsearch
network.host: 192.168.1.33

discovery.seed_hosts: ["192.168.1.31","192.168.1.32","192.168.1.33"]
cluster.initial_master_nodes: ["192.168.1.31","192.168.1.32","192.168.1.33"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#開啓master功能
node.master: true
#關閉ingest功能
node.ingest: false
#關閉data功能
node.data: false

elasticsearch集羣狀態

curl "http://192.168.1.31:9200/_cat/health?v"

elasticsearch-master狀態

curl "http://192.168.1.31:9200/_cat/nodes?v"

至此，當node.role裏全部服務器都再也不出現「mid」，則表示一切順利完成。

相關標籤/搜索

elasticsearch+elasticsearch

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。

EFK教程 - ElasticSearch角色分離

目錄

用途

架構

步驟說明

elasticsearch-data部署

elasticsearch-ingest部署

elasticsearch-master部署