轉載:Centos7.4安裝elasticsearch6.3+kibana6.3集羣

 

Centos7.4安裝elasticsearch+kibana集羣

 

 

主機環境

配置:css

節點數 4
操做系統 CentOS Linux release 7.4.1708 (Core)
內存 16GB

軟件環境

軟件 版本 下載地址
jdk jdk-8u172-linux-x64 點擊下載
elasticsearch elasticsearch-6.3.1 點擊下載
kibana kibana-6.3.1-linux-x86_64 點擊下載

主機規劃

4個節點角色規劃以下:html

主機名 pycdhnode1 pycdhnode2 pycdhnode3 pycdhnode4
IP 192.168.0.158 192.168.0.159 192.168.0.160 192.168.0.161
master節點 yes yes yes yes
data節點 yes yes yes yes
kibana yes no no no

注: 在實際生產中,仍是建議master節點和data節點分離java

主機安裝前準備

  1. 關閉全部節點的 SELinux
sed -i 's/^SELINUX=.*$/SELINUX=disabled/g' /etc/selinux/config setenforce 0 
  1. 關閉全部節點防火牆 firewalld or iptables
systemctl disable firewalld; systemctl stop firewalld; systemctl disable iptables; systemctl stop iptables; 
  1. 開啓全部節點時間同步 ntpdate
echo "*/5 * * * * /usr/sbin/ntpdate asia.pool.ntp.org | logger -t NTP" >> /var/spool/cron/root 
  1. 設置全部節點語言編碼以及時區
echo 'export TZ=Asia/Shanghai' >> /etc/profile echo 'export LANG=en_US.UTF-8' >> /etc/profile . /etc/profile 
  1. 全部節點添加elasticsearch用戶
useradd -m elasticsearch
echo 'elasticsearch' | passwd --stdin elasticsearch 

修改家目錄node

mv /home/elasticsearch /application chown -R elasticsearch. /application/elasticsearch 

vi /etc/passwd,修改elasticsearch用戶家目錄:python

elasticsearch:x:1001:1001::/application/elasticsearch:/bin/bash 

設置PS1linux

su - elasticsearch
echo 'export PS1="\u@\h:\$PWD>"' >> ~/.bash_profile echo "alias mv='mv -i' alias rm='rm -i'" >> ~/.bash_profile . ~/.bash_profile 
  1. 設置elasticsearch用戶之間免密登陸 首先在pycdhnode1主機生成祕鑰
su - elasticsearch ssh-keygen -t rsa # 一直回車便可生成elasticsearch用戶的公鑰和私鑰 cd .ssh vi id_rsa.pub # 去掉私鑰末尾的主機名 elasticsearch@pycdhnode1 cat id_rsa.pub > authorized_keys chmod 600 authorized_keys 

壓縮.ssh文件夾nginx

su - elasticsearch zip -r ssh.zip .ssh 

隨後分發ssh.zip到pycdhnode2-4主機elasticsearch用戶家目錄解壓即完成免密登陸git

  1. 主機內核參數優化以及最大文件打開數、最大進程數等參數優化 不一樣主機優化參數有可能不同,故這裏不做出具體優化方法,但若是elasticsearch環境用於正式生產,必須優化,linux默認參數可能會致使elasticsearch沒法啓動或者集羣性能低下。

注: 以上操做須要使用 root 用戶,到目前爲止操做系統環境已經準備完成,如下開始正式安裝,後面的操做若是不作特殊說明均使用 elasticsearch 用戶es6

安裝jdk1.8

全部節點都須要安裝,安裝方式都同樣 解壓 jdk-8u172-linux-x64.tar.gzgithub

tar zxvf jdk-8u172-linux-x64.tar.gz mkdir -p /application/elasticsearch/app mv jdk-8u172-linux-x64 /application/elasticsearch/app/jdk rm -f jdk-8u172-linux-x64.tar.gz 

配置環境變量 vi ~/.bash_profile 添加如下內容:

#java export JAVA_HOME=/application/elasticsearch/app/jdk export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin 

加載環境變量

. ~/.bash_profile 

查看是否安裝成功 java -version

java version "1.8.0_172" Java(TM) SE Runtime Environment (build 1.8.0_172-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) 

若是出現以上結果證實安裝成功。

安裝elasticsearch

首先在pycdhnode1上安裝 解壓 elasticsearch-6.3.1.tar.gz

tar zxvf elasticsearch-6.3.1.tar.gz mv elasticsearch-6.3.1 /application/elasticsearch/app/elasticsearch rm -f elasticsearch-6.3.1.tar.gz 

設置環境變量 vi ~/.bash_profile 添加如下內容:

#elasticsearch export ELASTICSEARCH_HOME=/application/elasticsearch/app/elasticsearch export PATH=$PATH:$ELASTICSEARCH_HOME/bin 

加載環境變量

. ~/.bash_profile 

添加配置文件 vi /application/elasticsearch/app/elasticsearch/config/elasticsearch.yml :

# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #cluster.name: py_es_6.3 # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: pyesnode-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 # # Set a custom port for HTTP: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true  #集羣的名稱 cluster.name: pyes6.3  #節點名稱,其他3個節點分別爲pyesnode-2,pyesnode-3,pyesnode-4 node.name: pyesnode-1  #指定該節點是否有資格被選舉成爲master節點,默認是true,es是默認集羣中的第一臺機器爲master,若是這臺機掛了就會從新選舉master node.master: true  #容許該節點存儲數據(默認開啓) node.data: true #實際生產能夠master節點和data數據分離  #索引數據的存儲路徑,多個目錄使用 , 分割 path.data: /application/elasticsearch/data/esdata  #日誌文件的存儲路徑 path.logs: /application/elasticsearch/app/elasticsearch/logs  #設置爲true來鎖住內存。由於內存交換到磁盤對服務器性能來講是致命的,當jvm開始swapping時es的效率會下降,因此要保證它不swap #bootstrap.memory_lock: true bootstrap.memory_lock: false #服務器內存小,設置容許使用swap  #綁定的ip地址 network.host: 0.0.0.0  #設置對外服務的http端口,默認爲9200 http.port: 9200  # 設置節點間交互的tcp端口,默認是9300 transport.tcp.port: 9300  #Elasticsearch將綁定到可用的環回地址,並將掃描端口9300到9305以嘗試鏈接到運行在同一臺服務器上的其餘節點。  #這提供了自動集羣體驗,而無需進行任何配置。數組設置或逗號分隔的設置。每一個值的形式應該是host:port或host #(若是沒有設置,port默認設置會transport.profiles.default.port 回落到transport.tcp.port)。 #請注意,IPv6主機必須放在括號內。默認爲127.0.0.1, [::1] discovery.zen.ping.unicast.hosts: ["pycdhnode1:9300", "pycdhnode2:9300", "pycdhnode3:9300", "pycdhnode4:9300"]  #若是沒有這種設置,遭受網絡故障的集羣就有可能將集羣分紅兩個獨立的集羣 - 分裂的大腦 - 這將致使數據丟失,通常設置(N/2)+1 discovery.zen.minimum_master_nodes: 3  #爲了使新加入的節點快速肯定master位置,能夠將data節點的默認的master發現方式有multicast修改成unicast:選擇性配置 #discovery.zen.ping.multicast.enabled: false #discovery.zen.ping.unicast.hosts: ["pycdhnode1", "pycdhnode2", "pycdhnode3", "pycdhnode4"] 
  • 其中的 node.name 配置每一個節點必須不同

設置節點內存使用量 vi /application/elasticsearch/app/elasticsearch/config/jvm.options

-Xms3g -Xmx3g 
  • 最小與最大必須設置同樣
  • 因爲jvm內存回收的緣由,當內存使用超過32G時,性能會下降,故每一個節點推薦最高設置31G
  • elasticsearch 2.x 版本設置內存使用在 $ELASTICSEARCH_HOME/bin/elasticsearch.in.sh中 ES_MIN_MEM=3g 與 ES_MAX_MEM=3g

建立所需目錄

mkdir -p /application/elasticsearch/data/esdata 

複製elasticsearch到pycdhnode2-4

scp ~/.bash_profile pycdhnode2:/application/elasticsearch scp ~/.bash_profile pycdhnode3:/application/elasticsearch scp ~/.bash_profile pycdhnode4:/application/elasticsearch scp -pr /application/elasticsearch/app/elasticsearch pycdhnode2:/application/elasticsearch/app scp -pr /application/elasticsearch/app/elasticsearch pycdhnode3:/application/elasticsearch/app scp -pr /application/elasticsearch/app/elasticsearch pycdhnode4:/application/elasticsearch/app ssh pycdhnode2 "mkdir -p /application/elasticsearch/data/esdata" ssh pycdhnode3 "mkdir -p /application/elasticsearch/data/esdata" ssh pycdhnode4 "mkdir -p /application/elasticsearch/data/esdata" 
  • 修改pycdhnode1-4 /application/elasticsearch/app/elasticsearch/config/elasticsearch.yml 中的 node.name pycdhnode1爲:pyesnode-1 ;pycdhnode2爲:pyesnode-2 ;pycdhnode3爲:pyesnode-3 ;pycdhnode4爲:pyesnode-4

優化全部主機參數,不然沒法啓動 vi /etc/sysctl.conf

vm.max_map_count=655360 

生效

sysctl -p 

vi /etc/security/limits.conf 添加如下內容:

* soft nofile 65536 * hard nofile 65536 * soft nproc 65536 * hard nproc 65536 

vi /etc/security/limits.d/20-nproc.conf 添加如下內容:

*          soft nproc 65536 root soft nproc unlimited 

重啓登陸 ulimit -a 查看是否生效

$ ulimit -a
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63488 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 

啓動elasticsearch 4個節點均啓動

/application/elasticsearch/app/elasticsearch/bin/elasticsearch -d 
  • -d 後臺服務的方式啓動
  • 若是啓動異常,查看日誌/application/elasticsearch/app/elasticsearch/logs/pyes6.3.log

查看進程

jps 

其中 Elasticsearch 進程即爲 elasticsearch

中止elasticsearch

kill pid 

查看集羣狀態

$ curl pycdhnode1:9200/_cat/health?v epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1531123674 16:07:54 pyes6.3 green 4 4 0 0 0 0 0 0 - 100.0% 
  • es 集羣一共3種狀態: green , yellow , red
  • 能夠看到集羣節點有4個,集羣狀態爲 green,正常

head插件安裝

ElasticSearch-head是一個H5編寫的ElasticSearch集羣操做和管理工具,能夠對集羣進行傻瓜式操做。

  • 顯示集羣的拓撲,而且可以執行索引和節點級別操做
  • 搜索接口可以查詢集羣中原始json或表格格式的檢索數據
  • 可以快速訪問並顯示集羣的狀態
  • 有一個輸入窗口,容許任意調用RESTful API。這個接口包含幾個選項,能夠組合在一塊兒以產生有趣的結果;
  • 5.0版本以前能夠經過plugin安裝,直接解壓即可運行,很綠色,5.0以後安裝就須要使用nodejs,而後以獨立服務的方式啓動,不太方便,能夠直接經過安裝谷歌瀏覽器插件 elasticsearch-head-chrome

首先在es集羣全部節點添加配置文件 vi /application/elasticsearch/app/elasticsearch/config/elasticsearch.yml

http.cors.enabled: true http.cors.allow-origin: "*" 

在pycdhnode1上面安裝,而後其餘主機能夠選裝,安裝方法同樣。

安裝NodeJS

wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.5.0-linux-x64.tar.gz tar zxvf node-v4.5.0-linux-x64.tar.gz mv node-v4.5.0-linux-x64 app/node rm -f node-v4.5.0-linux-x64.tar.gz 

添加環境變量 vi ~/.bash_profile

#node export NODE_HOME=/application/elasticsearch/app/node export PATH=$PATH:$NODE_HOME/bin export NODE_PATH=$NODE_HOME/lib/node_modules 

加載環境變量

. ~/.bash_profile 

安裝npm與grunt

npm install -g cnpm --registry=https://registry.npm.taobao.org npm install -g grunt npm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy 

下載head插件並安裝

wget  https://github.com/mobz/elasticsearch-head/archive/master.zip unzip master.zip mv elasticsearch-head-master app 

修改配置文件 vi /application/elasticsearch/app/elasticsearch-head-master/Gruntfile.js, 修改如下內容

connect: { server: { options: { hostname: '0.0.0.0', port: 9100, base: '.', keepalive: true } } } 
  • 能夠不修改,默認監聽9100

繼續編輯 vi /application/elasticsearch/app/elasticsearch-head-master/_site/app.js, 修改如下內容

this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://pycdhnode1:9200"; 
  • 如不修改,默認鏈接 http://pycdhnode1:9200,這裏能夠修改成集羣任一主機

下載依賴安裝

cd /application/elasticsearch/app/elasticsearch-head-master npm install 
  • 必須在head插件目錄中操做

啓動 head 插件 方法1:使用npm

cd /application/elasticsearch/app/elasticsearch-head-master npm run start 

方法2:直接使用grunt

cd /application/elasticsearch/app/elasticsearch-head-master grunt server 
  • 必須在head插件目錄中操做
  • npm 啓動方式本質上都是調用grunt啓動
  • 兩種啓動方式都不是後臺啓動,如需後臺運行,請使用nohup

訪問 head:

http://pycdhnode1:9100/

中止 head: 首先經過 ps aux|grep grunt 查找到進程 pid ,而後 kill pid

ElasticHQ管理工具安裝

ElasticHQ 是一款開源的具備良好體驗、直觀和功能強大的 ElasticSearch 的管理和監控工具。提供實時監控、全集羣管理、搜索和查詢,無需額外軟件安裝。最新版本支持ElasticSearch 2.x, 5.x, 6.x。 特色: 一、激活ES集羣和節點實時監控; 二、管理索引、分片、映射、別名、節點; 三、爲多個索引查詢提供查詢UI; 四、REST UI,不須要cURL和繁瑣的JSON格式; 五、100%基於瀏覽器,不需下載軟件; 六、免費;

ElasticHQ 是基於python的Django開發的,最新版本的安裝須要python3.4以上,安裝與啓動程序比較簡單,但要安裝python3.4以上環境比較麻煩,故咱們直接採用官方提供的docker容器安裝,簡單方便

首先在pull最新官方鏡像

docker pull elastichq/elasticsearch-hq 

啓動容器

docker run -d -p 9999:5000 --name es elastichq/elasticsearch-hq 

訪問

http://IP:9999

  • 打開首頁後在輸入框輸入es集羣隨意一臺節點地址確認便可

更多詳情參見:https://github.com/ElasticHQ/elasticsearch-HQ

kibana安裝

Kibana 是一個開源的分析和可視化平臺,旨在與 Elasticsearch 合做。Kibana 提供搜索、查看和與存儲在 Elasticsearch 索引中的數據進行交互的功能。開發者或運維人員能夠輕鬆地執行高級數據分析,並在各類圖表、表格和地圖中可視化數據。

kibana自己只提供單點安裝,若是想避免單點故障,須要結合lvs,haproxy,nginx等負載均衡軟件實現高可用,在這裏咱們 只在pycdhnode1上面安裝,而後其餘主機能夠選裝,安裝方法同樣。

安裝kibana

tar -zxvf kibana-6.3.1-linux-x86_64.tar.gz mv kibana-6.3.1-linux-x86_64 app/kibana rm -f kibana-6.3.1-linux-x86_64.tar.gz 

添加環境變量 vi ~/.bash_profile

#kibana export KIBANA_HOME=/application/elasticsearch/app/kibana export PATH=$PATH:$KIBANA_HOME/bin 

加載環境變量

. ~/.bash_profile 

配置文件 vi /application/elasticsearch/app/kibana/config/kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use. server.port: 5601 # 監聽端口  # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "0.0.0.0" # 監聽地址  # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: ""  # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they are rewritten by your reverse proxy. # This setting was effectively always `false` before Kibana 6.3 and will # default to `true` starting in Kibana 7.0. #server.rewriteBasePath: false  # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576  # The Kibana server's name. This is used for display purposes. server.name: "pycdhnode1"  # The URL of the Elasticsearch instance to use for all your queries. elasticsearch.url: "http://pycdhnode1:9200" # es鏈接地址,只能配置一個節點地址,若是須要高可用,須要es集羣配合lvs,haproxy負載均衡提供  # When this setting's value is true Kibana uses the hostname specified in the server.host # setting. When the value of this setting is false, Kibana uses the hostname of the host # that connects to this Kibana instance. #elasticsearch.preserveHost: true  # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. #kibana.index: ".kibana"  # The default application to load. #kibana.defaultAppId: "home"  # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. #elasticsearch.username: "user" #elasticsearch.password: "pass"  # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key  # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files validate that your Elasticsearch backend uses the same key files. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key  # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]  # To disregard the validity of SSL certificates, change this setting's value to 'none'. #elasticsearch.ssl.verificationMode: full  # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500  # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000  # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ]  # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {}  # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000  # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying. #elasticsearch.startupTimeout: 5000  # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false  # Specifies the path where Kibana creates the process ID file. #pid.file: /var/run/kibana.pid  # Enables you specify a file where Kibana stores log output. #logging.dest: stdout  #Set the value of this setting to true to suppress all logging output. #logging.silent: false  #Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false  #Set the value of this setting to true to log all events, including system usage information #and all requests. #logging.verbose: false  #Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000  # The default locale. This locale can be used in certain circumstances to substitute any missing # translations. #i18n.defaultLocale: "en" xpack.security.enabled: false #關閉xpack驗證;因爲集羣爲配置xpack,故必須關閉,不然沒法正常鏈接es集羣 

啓動 kibana 方法1:控制檯啓動

kibana 
  • 退出回話或者 ctrl + c 會退出

方法2:使用nohup後臺啓動

cd /application/elasticsearch/app/kibana mkdir logs nohup kibana > logs/server.log 2>&1 & 

訪問 kibana:

http://pycdhnode1:5601/

中止 kibana: 首先經過 ps aux|grep kibana 查找到進程 pid ,而後 kill pid

更多kibana使用方法參考官網:https://www.elastic.co/guide/en/kibana/6.3/index.html

相關文章
相關標籤/搜索