生產環境elasticsearch5.0.1和6.3.2集羣的部署配置詳解

線上環境elasticsearch5.0.1集羣的配置部署

es集羣的規劃：
硬件：
7臺8核、64G內存、2T ssd硬盤加1臺8核16G的阿里雲服務器

其中一臺做爲kibana+kafka鏈接查詢的服務器
其餘6臺都做爲node和master兩種角色

操做系統：centos7.2 x86_64
爲方便磁盤擴容建議將磁盤進行lvm邏輯卷配置，能夠參考：
aliyun添加數據盤後的物理分區和lvm邏輯卷兩種掛載方式
http://blog.csdn.net/reblue520/article/details/54174178

1.安裝jdk1.8和elasticsearch5.0.1

rpm -ivh jdk-8u111-linux-x64.rpm
tar -zxvf elasticsearch-5.0.1.tar.gz

2.添加yunva這個運行elasticsearch的用戶(es必須使用非root用戶啓動)

useradd yunva -d /home/yunva
echo 'pass'|passwd --stdin yunva

chown -R yunva.yunva /data

修改默認端口
sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
sed -i 's/^GSSAPIAuthentication yes$/GSSAPIAuthentication no/' /etc/ssh/sshd_config
service sshd restart

3.針對es作的一些系統的優化配置

swapoff -a

echo "fs.file-max = 1000000" >> /etc/sysctl.conf
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
echo "vm.swappiness = 1" >> /etc/sysctl.conf

sysctl -p
sed -i 's/* soft nofile 65535/* soft nofile 655350/g' /etc/security/limits.conf
sed -i 's/* hard nofile 65535/* hard nofile 655350/g' /etc/security/limits.conf

將java_home加入環境變量
cat >> /etc/profile <<EOF
export JAVA_HOME=/usr/java/jdk1.8.0_111 
export PATH=\$JAVA_HOME/bin:\$PATH
EOF

source /etc/profile

4.es內存調整配置文件(建議配置爲物理內存的一半或者更多最好不要超過32G，超過了也可能不會加強性能)：

/data/elasticsearch-5.0.1/config/jvm.options

sed -i 's/-Xms2g/-Xms32g/' /data/elasticsearch-5.0.1/config/jvm.options
sed -i 's/-Xmx2g/-Xmx32g/' /data/elasticsearch-5.0.1/config/jvm.options
echo "-Xss256k" >>/data/elasticsearch-5.0.1/config/jvm.options

sed -i 's/-XX:+UseConcMarkSweepGC/-XX:+UseG1GC/' /data/elasticsearch-5.0.1/config/jvm.options

5.集羣的主要配置文件

修改elasticsearch的參數
vim /etc/elasticsearch/elasticsearch.yml(rpm安裝方式的配置文件位置)
vim  /data/elasticsearch-5.0.1/config/elasticsearch.yml

es節點的配置：
# 節點名
cluster.name: yunva-es
# 集羣的名稱，能夠不寫
discovery.zen.ping.unicast.hosts: ["node-1","yunva_etl_es2", "yunva_etl_es3","yunva_etl_es4","yunva_etl_es5","yunva_etl_es6","yunva_etl_es7"]
node.name: yunva_etl_es6
node.master: true
node.data: true
path.data: /data/es/data
path.logs: /data/es/logs
action.auto_create_index: false
indices.fielddata.cache.size: 12g
bootstrap.memory_lock: false
# 內網地址，能夠加快速度
network.host: 192.168.1.10
http.port: 9200
# 增長新的參數head插件能夠訪問es
http.cors.enabled: true
http.cors.allow-origin: "*"

gateway.recover_after_time: 8m
gateway.expected_nodes: 3
cluster.routing.allocation.node_initial_primaries_recoveries: 8

# 如下配置能夠減小當es節點短期宕機或重啓時shards從新分佈帶來的磁盤io讀寫浪費
discovery.zen.fd.ping_timeout: 180s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 120s

針對kibana的es配置(非node和master節點)
# cat /etc/elasticsearch/elasticsearch.yml
cluster.name: yunva-es
node.name: yunva_etl_es1
node.master: false
node.data: false
node.ingest: false

action.auto_create_index: false
path.data: /data/es/data
path.logs: /data/es/logs
bootstrap.memory_lock: false
network.host: 0.0.0.0
http.port: 9200

http.cors.enabled: true
http.cors.allow-origin: "*"

# 如下配置能夠減小當es節點短期宕機或重啓時shards從新分佈帶來的磁盤io讀寫浪費
discovery.zen.fd.ping_timeout: 180s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 120s

注意修改配置文件vim /etc/hosts 列出集羣節點名稱和對應ip地址的對應關係(有內網dns而且配置的就不須要再次配置了)

echo "10.28.50.131 node-1" >> /etc/hosts
echo "10.26.241.239 yunva_etl_es3" >> /etc/hosts
echo "10.25.135.215 yunva_etl_es2" >> /etc/hosts
echo "10.26.241.237 yunva_etl_es4" >> /etc/hosts
echo "10.27.78.228 yunva_etl_es5" >> /etc/hosts
echo "10.27.65.121 yunva_etl_es6" >> /etc/hosts
echo "10.27.35.94 yunva_etl_es7" >> /etc/hosts

6.建立日誌和數據存放目錄

mkdir -p /data/es/data
mkdir /data/es/logs
chown -R yunva.yunva /data

7.啓動es服務：

# su - yunva
[yunva]$ cd /data/elasticsearch-5.0.1/bin/
./elasticsearch &

8.檢查單臺服務是否正常：

$ curl http://ip:9200/
{
  "name" : "yunva_etl_es5",
  "cluster_name" : "yunva-es",
  "cluster_uuid" : "2shAg8u3SjCRNJ4mEUBzBQ",
  "version" : {
    "number" : "5.0.1",
    "build_hash" : "080bb47",
    "build_date" : "2016-11-11T22:08:49.812Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.1"
  },
  "tagline" : "You Know, for Search"
}


# 查看集羣狀態
$ curl http://ip:9200/_cluster/health/?pretty
{
  "cluster_name" : "yunva-es",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 66,
  "active_shards" : 132,
  "relocating_shards" : 2,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}


而後將配置好的es程序拷貝到其餘服務器中，注意修改如下內容(network.host爲內網地址，速度更快，節省互相複製、分片的時候處理帶寬)：
1.elasticsearch.yml文件的配置修改
node.name: 節點名稱
network.host: es節點的內網IP地址
2./etc/hosts文件中內網ip和node.name的對應關係


後續添加對集羣服務的監控，能夠參考：

 

elasticsearch6.0的安裝

1、肯定服務器配置，新建硬件資源

須要配置es集羣做爲熱數據，數據量不大，兩臺8核16G，200G磁盤的服務器作es集羣便可


2、初始化系統

1、關閉firewall，並安裝iptables服務
systemctl stop firewalld.service #中止firewall

systemctl disable firewalld.service #禁止firewall開機啓動

yum install iptables-services -y


# 添加防火牆規則
vim /usr/local/worksh/FireWall.sh


#!/bin/bash
iptables -A INPUT -p udp --dport 1701 -j ACCEPT
################################################################
### Required modules
/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe iptable_mangle
/sbin/modprobe iptable_nat
/sbin/modprobe ipt_LOG
/sbin/modprobe ipt_limit
/sbin/modprobe ipt_state
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_nat_ftp
/sbin/modprobe ipt_owner
/sbin/modprobe ipt_REJECT

### Clean Rules
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -P PREROUTING ACCEPT
iptables -t nat -P POSTROUTING ACCEPT
iptables -t nat -P OUTPUT ACCEPT
iptables -t mangle -P PREROUTING ACCEPT
iptables -t mangle -P OUTPUT ACCEPT
iptables -F
iptables -t nat -F
#iptables -t mangle -F
iptables -X
iptables -t nat -X
#iptables -t mangle -X

### Drop all pocket,first
iptables -P INPUT DROP
#iptables -P OUTPUT DROP
iptables -P FORWARD DROP

### Create New chains
iptables -N bad_tcp_packets
#iptables -N allowed
iptables -N icmp_packets

### Bad_tcp_packets chain
/sbin/iptables -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags ALL ALL        -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags ALL NONE         -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags SYN,FIN SYN,FIN  -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags SYN,RST SYN,RST  -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags FIN,RST FIN,RST  -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags ACK,FIN FIN      -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags ACK,PSH PSH      -j DROP
/sbin/iptables -A bad_tcp_packets -p tcp --tcp-flags ACK,URG URG      -j DROP

### ICMP Rules
iptables -A icmp_packets -p icmp --icmp-type 8 -j ACCEPT
iptables -A icmp_packets -p icmp --icmp-type 11 -j ACCEPT
#iptables -A icmp_packets -p icmp -j DROP

##keepalived
iptables -A INPUT -i eth1 -p vrrp -s 63.159.217.139 -j ACCEPT

### LookBack and Private interface
iptables -A INPUT -p ALL -i lo -j ACCEPT
iptables -A INPUT -p ALL -i eth0 -j ACCEPT

### INPUT chain
iptables -A INPUT -p tcp -j bad_tcp_packets
iptables -A INPUT -p icmp -j icmp_packets
iptables -A INPUT -p ALL -m state --state ESTABLISHED,RELATED -j ACCEPT

### 5666 EOP
iptables -A INPUT -p tcp -i eth1 --dport 5666 -s 47.91.75.80 -j ACCEPT

##rep
iptables -A INPUT -p tcp -i eth1 --dport 3306 -s 47.91.90.28 -j ACCEPT


# Count Limit
#iptables -A INPUT -m limit --limit 3/minute --limit-burst 3 -j LOG --log-level INFO --log-prefix "IPT INPUT PACKET DIED:"
############################################################
iptables -I INPUT  -p udp --dport 1701 -j ACCEPT

### Open Ports
Public_access="80 443 8080"
Server_access="20 21 873 3306 2008 8899 9200 9300"
Company_access="20 21 873 9200 9300 9100"
aliyun_access="3306"
### Allow Ips

Servers_ip="192.168.254.0/24"
Company_ip="1.1.1.1"
aliyun_ip="2.2.2.2"
Eus_server_ip="3.3.3.3"

### Public access Rules
for port in $Public_access
do
  iptables -A INPUT -p tcp -i eth1 --dport $port -j ACCEPT
done

### Servers access Rules
for port in $Server_access
do
  for ip in $Servers_ip
  do
    iptables -A INPUT -p tcp -i eth1 --dport $port -s $ip -j ACCEPT
  done
done

### Eus_server access Rules
for port in $Server_access
do
  for ip in $Eus_server_ip
  do
    iptables -A INPUT -p tcp -i eth1 --dport $port -s $ip -j ACCEPT
  done
done

### Company access Rules
for port in $Company_access
do
  for ip in $Company_ip
  do
    iptables -A INPUT -p tcp -i eth1 --dport $port -s $ip -j ACCEPT
  done
done

# chmod +x /usr/local/worksh/FireWall.sh
# sh -x /usr/local/worksh/FireWall.sh


# 開機運行防火牆規則
[root@cms_elasticsearch01 elasticsearch-head]# tail -n 1 /etc/rc.local 
/usr/local/worksh/FireWall.sh

*********************

# centos7設置洛杉磯時間

timedatectl set-timezone America/Los_Angeles

# 修改文件句柄
sed -i 's#root soft nofile 65535#root soft nofile 655350#g' /etc/security/limits.conf
sed -i 's#root hard nofile 65535#root hard nofile 655350#g' /etc/security/limits.conf
sed -i 's#* soft nofile 65535#* soft nofile 655350#g' /etc/security/limits.conf
sed -i 's#* hard nofile 65535#* hard nofile 655350#g' /etc/security/limits.conf
sed -i 's#*          soft    nproc     4096##g' /etc/security/limits.d/20-nproc.conf

# 最終效果
root soft nofile 655350
root hard nofile 655350
* soft nofile 655350
* hard nofile 655350

# 安裝依賴
yum install -y gcc gcc-c++ htop telnet iotop iptraf iftop make logrotate xinetd ntsysv  sysstat perl autoconf libjpeg libjpeg-devel libpng libpng-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel e2fsprogs e2fsprogs-devel krb5-devel libidn libidn-devel libxml2 libxml2-devel libxslt-devel libevent-devel libtool libtool-ltdl bison  ntpdate  patch vim wget openssh-clients bind-utils

# 系統升級到最新
yum update -y


# 安裝jdk1.8_181

rpm -ivh jdk-8u181-linux-x64.rpm
# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
export PATH=$JAVA_HOME/bin:$PATH

4.添加運行elasticsearch的普通用戶
##### 安裝elasticsearch

elasticsearch 
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz
tar -zxf elasticsearch-6.3.2.tar.gz

#### 建立elasticsearch的普通帳號
groupadd elasticsearch
useradd elasticsearch -g elasticsearch
chown -R elasticsearch.elasticsearch /data
chown -R elasticsearch.elasticsearch /usr/local/elasticsearch-6.3.2/

# 一些優化配置
swapoff -a

echo "fs.file-max = 1000000" >> /etc/sysctl.conf
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
echo "vm.swappiness = 1" >> /etc/sysctl.conf
# 讓配置生效
sysctl -p

# 建立數據和日誌目錄
mkdir -p /data/es/data
mkdir -p /data/es/logs

es節點的配置：

主節點 master 配置：
[root@cms_elasticsearch01 config]# cat elasticsearch.yml
# 節點名
cluster.name: cms-uat-es
# 集羣的名稱，能夠不寫
discovery.zen.ping.unicast.hosts: ["cms_elasticsearch01","cms_elasticsearch02"]
node.name: cms_elasticsearch01
node.master: true
node.data: true
path.data: /data/es/data
path.logs: /data/es/logs
#action.auto_create_index: false
indices.fielddata.cache.size: 1g
# 內網地址，能夠加快速度
#network.host: 192.168.254.36
network.host: 0.0.0.0
http.port: 9200
# 增長新的參數head插件能夠訪問es
http.cors.enabled: true
http.cors.allow-origin: "*"

gateway.recover_after_time: 8m

# 如下配置能夠減小當es節點短期宕機或重啓時shards從新分佈帶來的磁盤io讀寫浪費
discovery.zen.fd.ping_timeout: 300s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 180s

數據節點配置
[root@cms_elasticsearch02 ~]# cat /usr/local/elasticsearch-6.3.2/config/elasticsearch.yml
# 節點名
cluster.name: cms-uat-es
# 集羣的名稱，能夠不寫
discovery.zen.ping.unicast.hosts: ["cms_elasticsearch01","cms_elasticsearch02"]
node.name: cms_elasticsearch02
node.master: false
node.data: true
path.data: /data/es/data
path.logs: /data/es/logs
#action.auto_create_index: false
indices.fielddata.cache.size: 1g
bootstrap.memory_lock: false
# 內網地址，能夠加快速度
#network.host: 192.168.254.37
network.host: 0.0.0.0
http.port: 9200
# 增長新的參數head插件能夠訪問es
http.cors.enabled: true
http.cors.allow-origin: "*"

gateway.recover_after_time: 8m

# 如下配置能夠減小當es節點短期宕機或重啓時shards從新分佈帶來的磁盤io讀寫浪費
discovery.zen.fd.ping_timeout: 300s
discovery.zen.fd.ping_retries: 8
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 180s

# 修改hosts文件
echo "192.168.254.36 cms_elasticsearch01" >> /etc/hosts
echo "192.168.254.37 cms_elasticsearch02" >> /etc/hosts


# 啓動程序
su elasticsearch -c "cd /usr/local/elasticsearch-6.3.2 && bin/elasticsearch -d"

3、添加zabbix監控
硬件資源的監控
es的監控


# 集羣狀態監控腳本
UserParameter=es_status,curl -sXGET http://192.168.254.37:9200/_cluster/health/?pretty | grep "status"|awk -F '[ "]+' '{print $4}'|grep -c 'green'

# 若是出現錯誤的監控
UserParameter=es_debug,sudo /bin/find /usr/local/elasticsearch-6.3.2 -name hs_err_pid*.log -o -name java_pid*.hprof|wc -l

# 監控觸發自動啓動elasticsearch腳本：

# vim /usr/local/zabbix_agents_3.2.0/scripts/start_es.sh

#!/bin/bash
# if elasticsearch process exists kill it
source /etc/profile

count_es=`ps -ef|grep elasticsearch|grep -v grep|wc -l`
if [ $count_es -ge 1 ];then
ps -ef|grep elasticsearch|grep -v grep|/bin/kill `awk '{print $2}'`
fi
rm -f /usr/local/elasticsearch-6.3.2/bin/java_pid*.hprof
# start it
su elasticsearch -c "cd /usr/local/elasticsearch-6.3.2 && bin/elasticsearch -d"


# chmod +x /usr/local/zabbix_agents_3.2.0/scripts/start_es.sh


#########################################


# 安裝head插件
# yum install -y epel-release
# yum install -y nodejs

[root@cms_elasticsearch01 elasticsearch-head]# npm install -g cnpm --registry=https://registry.npm.taobao.org
npm WARN deprecated socks@1.1.10: If using 2.x branch, please upgrade to at least 2.1.6 to avoid a serious bug with socket data flow and an import issue introduced in 2.1.0
/usr/bin/cnpm -> /usr/lib/node_modules/cnpm/bin/cnpm
/usr/lib
└─┬ cnpm@6.0.0 
  ├── auto-correct@1.0.0 
  ├── bagpipe@0.3.5 

[root@cms_elasticsearch01 elasticsearch-head]# npm install -g grunt
/usr/bin/grunt -> /usr/lib/node_modules/grunt/bin/grunt
/usr/lib
└─┬ grunt@1.0.3 

[root@cms_elasticsearch01 elasticsearch-head]# npm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy
/usr/bin/grunt -> /usr/lib/node_modules/grunt-cli/bin/grunt
/usr/lib
└─┬ grunt-cli@1.3.1


[root@cms_elasticsearch01 elasticsearch-head]# grunt -version
grunt-cli v1.3.1


（6）下載依賴 
進入elasticsearch-head-master目錄，執行下面命令
[root@cms_elasticsearch01 elasticsearch-head]# npm install


(7)修改配置，當有外網和內網IP的時候須要修改配置
 elasticsearch-head/Gruntfile.js 文件
    connect: {
      server: {
        options: {
          hostname: '*',
          port: 9100,
          base: '.',
          keepalive: true

# 修改 elasticsearch-head/_site/app.js 將localhost修改成本機外網ip地址
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://70.39.189.23:9200";

# 防火牆要互相放行
# 啓動head插件
su elasticsearch -c "cd /usr/local/elasticsearch-head && nohup npm run start >> /usr/local/elasticsearch-head/logs/npm-start.log 2>&1 &"

# chown -R elasticsearch.elasticsearch /usr/local/elasticsearch-head/


# 啓動head插件腳本

#!/bin/sh
# stop elasticsearch master function
function stopelkmaster()
{
        count=`ps -ef | grep -v grep |grep grunt | wc -l`
        if [ $count -gt 0 ];then
                for i in `ps -ef | grep -v grep |grep grunt|awk '{print $2}'`
                do
                        #echo "is exists:"
                        kill -9 $i
                done
        fi
}

# start elasticsearch master function
function startelkmaster()
{
        count=`ps -ef | grep -v grep |grep grunt | wc -l`
        if [ $count -gt 0 ];then
                for i in `ps -ef | grep -v grep |grep grunt|awk '{print $2}'`
                do
                        kill -9 $i
                done
                sleep 1
                cd /opt/elasticsearch-head
                su elasticsearch -c "cd /usr/local/elasticsearch-head && nohup npm run start >> /usr/local/elasticsearch-head/logs/npm-start.log 2>&1 &"
        else
                cd /opt/elasticsearch-head
                su elasticsearch -c "cd /usr/local/elasticsearch-head && nohup npm run start >> /usr/local/elasticsearch-head/logs/npm-start.log 2>&1 &"
        fi
}
isExistsElk=`ps -ef | grep -v grep |grep grunt | wc -l`
if [ $isExistsElk -eq 0 ]
then
        #echo "start process....."
        startelkmaster
elif [ $isExistsElk -gt 0 ]
then
        stopelkmaster
        sleep 1
        startelkmaster
else
  echo "error!"
fi