ES搜索引擎集羣模式搭建【Kibana可視化】

一.簡介

　　ElasticSearch是一個基於Lucene的搜索服務器。它提供了一個分佈式多用戶能力的全文搜索引擎（與Solr相似），基於RESTful web接口。Elasticsearch是用Java開發的，並做爲Apache許可條款下的開放源碼發佈，是當前流行的企業級搜索引擎。設計用於雲計算中，可以達到實時搜索，穩定，可靠，快速，安裝使用方便。

二.相關概念

　　cluster

　　表明一個 集羣，集羣中有多個節點，其中有一個爲主節點，這個主節點經過選舉產生，主從節點是對於集羣內部來講的。ES的一個概念就是去中心化，字面上理解就是無中心節點，這是對於集羣外部來講的，由於從外部來看ES集羣，在邏輯上是個總體，你與任何一個節點的通訊和與整個ES集羣通訊是等價的。

　　shards

　　表明索引分片，ES能夠把一個完整的索引分紅多個分片，這樣的好處是能夠把一個大的索引拆分紅多個，分佈到不一樣的節點上。構成分佈式搜索。分片的數量只能在索引建立前指定，而且索引建立後不能更改。

　　replicas

　　表明索引副本，ES能夠設置多個索引的副本，副本的做用一是提升系統的 容錯性，當某個節點某個分片損壞或丟失時能夠從副本中恢復。二是提升ES的查詢效率，ES會自動對搜索請求進行負載均衡。

　　recovery

　　表明數據恢復或叫數據從新分佈，es在有節點加入或退出時會根據機器的負載對索引分片進行從新分配，掛掉的節點從新啓動時也會進行數據恢復。

　　river

　　表明es的一個數據源，也是其它存儲方式（如：數據庫）同步數據到es的一個方法。它是以插件方式存在的一個es服務，經過讀取river中的數據並把它索引到es中，官方的river有couchDB的，RabbitMQ的，Twitter的，Wikipedia的。

　　gateway

　　表明es索引快照的存儲方式，es默認是先把索引存放到內存中，當內存滿了時再持久化到本地硬盤。gateway對索引快照進行存儲，當這個es集羣關閉再從新啓動時就會從gateway中讀取索引備份數據。es支持多種類型的gateway，有本地文件系統（默認）， 分佈式文件系統，Hadoop的HDFS和amazon的s3 雲存儲服務。

　　discovery.zen

　　表明es的自動發現節點機制，es是一個基於p2p的系統，它先經過廣播尋找存在的節點，再經過 多播協議來進行節點之間的通訊，同時也支持 點對點的交互。

　　Transport

　　表明es內部節點或集羣與客戶端的交互方式，默認內部是使用tcp協議進行交互，同時它支持http協議（json格式）、 thrift、servlet、memcached、zeroMQ等的 傳輸協議（經過 插件方式集成）。

三.上傳

 　 

　　

四.建立用戶

　　

五.配置

　　修改配置文件：elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: zhen-es # 集羣名稱
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1 # 節點名稱
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.245.133 # 本機ip
#
# Set a custom port for HTTP:
#
http.port: 9200 # 開放端口
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
# 配置放置腦裂 # discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["192.168.245.130","192.168.245.131", "192.168.245.133"] discovery.zen.ping_timeout: 120s client.transport.ping_timeout: 60s

六.分發到其它節點（根據節點進行相應的修改）

　　執行命令：scp -r ./elasticsearch-6.6.0/ root@worker1:/usr/local/soft/

　　執行命令：scp -r ./elasticsearch-6.6.0/ root@worker2:/usr/local/soft/

　　

　　修改權限

　　

　　修改相應配置

　　節點worker1:

　　　　node.name: node-2

　　　　network.host: 192.168.245.130

　　節點worker2:

　　　　node.name: node-3

　　　　network.host: 192.168.245.131

七.測試ES

　　1.切換用戶

　　　　su es

　　2.執行

　　　　cd ..

　　　　cd bin

　　　　./elasticsearch

　　3.結果

　　　　錯誤一

　　　　

　　　　解決方案一

　　　　編輯/etc/security/limits.conf，添加

　　　　* soft nofile 65536

　　　　* hard nofile 131072

　　　　* soft nproc 2048
　　　　* hard nproc 4096

　　　　節點master：

　　　　　

　　　　錯誤二

　　　　

　　　　第二個錯誤是新出現的，解決方案二

　　　　修改：/etc/security/limits.d/20-nproc.conf 

　　　　修改：4096-》5120

　　　　節點worker1:

　　　　

　　　　節點worker2:

　　　　

八.配置可視化顯示和交互平臺Kibana

　　1.簡介

　　　　Kibana 是一款開源的數據分析和可視化平臺，它是 Elastic Stack 成員之一，設計用於和 Elasticsearch 協做。您可使用 Kibana 對Elasticsearch索引中的數據進行搜索、查看、交互操做。您能夠很方便的利用圖表、表格及地圖對數據進行多元化的分析和呈現。

　　　　Kibana 可使大數據通俗易懂。它很簡單，基於瀏覽器的界面便於您快速建立和分享動態數據儀表板來追蹤 Elasticsearch 的實時數據變化。

 　　2.配置

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "192.168.245.133" # 根據節點ip配置

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.245.130:9200","http://192.168.245.131:9200","http://192.168.245.133:9200"] # 配置集羣中全部的節點

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
#i18n.locale: "en"

　　3.啓動

　　　　運行命令：./kibana

　　4.查看ES集羣狀況

　　以及節點詳情：

　　此時表示ES集羣配置及啓動成功！