ElasticSearch

ElasticSearch部署服務與配置

項目中遇到了須要大數據量的查詢需求，本來使用的PostgreSQL在查詢性能上不符合要求，因而就使用了ElasticSearch

ElasticSearch的底層是使用java開發的，因此須要安裝java的JDK

 

安裝java
yum install java-1.8.0-openjdk*

配置環境變量
vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el6_10.x86_64
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

使配置文件生效：source /etc/profile

驗證java版本
java -version

 

 

 

開始是將ElasticSearch經過drf-haystack集成到django項目中，過程當中發現這個庫由於elasticsearch在2.x之後的版本中棄用了一個變量，在5.x後的版本刪除了這個變量，2.x之後的版本會報警告，5.x之後的版本會報錯誤，致使只能完美支持1.x的版本，因此決定將ElasticSearch獨立成單獨的服務，前端使用dsl語言直接訪問服務進行查詢，後臺使用elasticsearch的庫進行插入數據，這裏實在CentOS6.9下使用的6.7.1版本，你們能夠選用適合本身的版本，下載地址：https://www.elastic.co/cn/downloads/elasticsearch

1.ElasticSearch不能夠用root用戶來啓動，這裏建立了一個用戶

groupadd esuser
useradd esuser -g esuser -p esuser12345

修改密碼 Lantucx2018
passwd esuser

2.將下載下來的壓縮包拷貝到esuser的目錄下，更改文件的權限，使esuser能夠有權限操做Elasticsearch

tar -xzvf elasticsearch-6.7.1.tar.gz

chown -R esuser:esuser elasticsearch-6.7.1

3.修改配置文件

vim  elasticsearch-6.7.1/config/elasticsearch.yml
ip改成 192.168.1.110
將端口號改成 8200

注意要在Memory下面: 
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false

#開啓跨域訪問支持，默認爲false
http.cors.enabled: true
http.cors.allow-origin: "*"

node.max_local_storage_nodes: 100

#或跨域訪問容許的域名地址，(容許全部域名)以上使用正則
http.cors.allow-origin: /.*/ 

修改heap_size:
vim elasticsearch-6.7.1/config/jvm.options
修改容許使用的內存
-Xms32g
-Xmx32g

4.切換用戶，開啓ElasticSearch

su esuser
cd elasticsearch-6.7.1/bin
./elasticsearch

下面是我遇到的一些錯誤：

若是報錯以下:
[2019-04-18T19:32:06,924][INFO ][o.e.t.TransportService   ] [YWePpHM] publish_address {192.168.164.176:9300}, bound_addresses {192.168.164.176:9300}
[2019-04-18T19:32:06,964][INFO ][o.e.b.BootstrapChecks    ] [YWePpHM] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max number of threads [1024] for user [elsearch] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2019-04-18T19:32:07,039][INFO ][o.e.n.Node               ] [YWePpHM] stopping ...
[2019-04-18T19:32:07,110][INFO ][o.e.n.Node               ] [YWePpHM] stopped
[2019-04-18T19:32:07,110][INFO ][o.e.n.Node               ] [YWePpHM] closing ...
[2019-04-18T19:32:07,153][INFO ][o.e.n.Node               ] [YWePpHM] closed
[2019-04-18T19:32:07,159][INFO ][o.e.x.m.p.NativeController] [YWePpHM] Native controller process has stopped - no new native processes can be started


解決：
5.1、文件句柄不足
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
su root 
cp /etc/security/limits.conf /etc/security/limits.conf.bak
vi /etc/security/limits.conf
備註：使用最高權限 修改安全配置 在文件末尾加入
# End of file
esuser       hard        nofile        65536 
esuser       soft        nofile        65536
*            soft        nproc         4096
*            hard        nproc         4096 

備註： esuser爲用戶名 能夠是使用*進行通配  
nofile 最大打開文件數目

5.二、啓動最大線程數限制
[2]: max number of threads [3802] for user [esuser] is too low, increase to at least [4096]
切換到root用戶下
cd /etc/security/limits.d/
cp 90-nproc.conf 90-nproc.conf.bak

修改90-nproc.conf（系統不一樣文件名有變化）：
vim /etc/security/limits.d/90-nproc.conf
修改 * soft nproc 1024爲
* soft nproc 4096

5.4.設定虛存不足
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
step1:修改安全限制配置文件
cp /etc/sysctl.conf /etc/sysctl.conf.bak
vim /etc/sysctl.conf

備註：行末加上vm.max_map_count = 262144
保存退出
使配置生效：sysctl -p

以下：
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

vm.max_map_count = 262144
備註：vm.max_map_count = 262144 值大於錯誤提示值65530

5.五、
ERROR: [1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
Centos6不支持SecComp，而ES5.2.0後的版本默認bootstrap.system_call_filter爲true，在elasticsearch.yml中增長bootstrap.system_call_filter爲false，注意要在Memory下面:
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false

5.六、啓動
su esuser
/elasticsearch-6.3.0/bin/elasticsearch -d -p pid

服務搭建好之後這裏使用python的elasticsearch庫導入的數據，推薦一個谷歌插件，用來查看Es服務，查詢索引狀態，構建查詢條件都比較方便，叫elasticsearch-head