Elasticsearch6.二、head插件、x-pack安全模塊(security機制)安裝

ElasticSearch簡介

a.ElasticSearch是一個基於Lucene開發的搜索服務器，具備分佈式多用戶的能力，ElasticSearch是用Java開發的開源項目(Apache許可條款)，基於Restful Web接口，可以達到實時搜索、穩定、可靠、快速、高性能、安裝使用方便，同時它的橫向擴展能力很是強，不須要重啓服務。
b.ElasticSearch是一個很是好用的實時分佈式搜索和分析引擎，能夠幫助咱們快速的處理大規模數據，也能夠用於全文檢索，結構化搜索以及分析等。
c.目前不少網站都在使用ElasticSearch進行全文檢索，例如：GitHub、StackOverflow、Wiki等。
d.ElasticSearch式創建在全文檢索引擎Lucene基礎上的，而Lucene是最早進、高效的開元搜索引擎框架，,它可使你的引用程序添加索引和搜索能力，可是Lucene只是一個框架，要充分利用它的功能，咱們須要很高的學習成本，而ElasticSearch使用Lucene做爲內部引擎，在其基礎上封裝了功能強大的Restful API，讓開發人員能夠在不須要了解背後複雜的邏輯，便可實現比較高效的搜索。

• ElasticSearch安裝
  Elasticsearch是基於java寫的，因此它的運行環境中須要java的支持。

  [root@localhost 40]# java -version
  -bash: java: command not found
  [root@localhost 40]# wget http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz
  [root@localhost 40]# tar xvf jdk-8u161-linux-x64.gz
  [root@localhost 40]# vim /etc/profile
  export JAVA_HOME=/opt/40/jdk1.8.0_161
  export JAVA_BIN=/opt/40/jdk1.8.0_161/bin
  export PATH=$PATH:$JAVA_HOME/bin
  export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  export JAVA_HOME JAVA_BIN PATH CLASSPATH
  [root@localhost 40]# source /etc/profile
  [root@localhost 40]# java -version
  java version "1.8.0_161"
  Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
  Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
  [root@localhost 40]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.0.tar.gz
  [root@localhost 40]# tar zxvf elasticsearch-6.2.0.tar.gz

* ElasticSearch啓動

[root@localhost 40]# cd elasticsearch-6.2.0/bin/
[2018-02-05T08:29:10,244][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.0.jar:6.2.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.0.jar:6.2.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.0.jar:6.2.0]
... 6 more
[root@localhost bin]#

當使用root帳戶啓動ES會出現以上錯誤信息，這是由於處於系統安裝考慮的設置，因爲Elasticsearch能夠接收用戶輸入的腳本而且執行，爲了系統安全考慮，不容許root帳號啓動，因此建議給Elasticsearch單首創建一個用戶來運行Elasticsearch。

[root@localhost bin]# groupadd justin
[root@localhost bin]# useradd justin -g justin -p 51cto
[root@localhost bin]# chown -R justin:justin /opt/40/elasticsearch-6.2.0/
[root@localhost bin]# su - justin
[justin@localhost ~]$ cd /opt/40/elasticsearch-6.2.0/bin/
[justin@localhost bin]$ cp ../config/elasticsearch.yml{,.default}
[justin@localhost bin]$ vim ../config/elasticsearch.yml
54 #
55 network.host: 0.0.0.0
56 #
57 # Set a custom port for HTTP:
58 #
59 http.port: 9200
60 #
[justin@localhost bin]$ ./elasticsearch -d
[justin@localhost bin]$ ps -ef|grep -i elas
justin 13348 1 38 08:34 pts/2 00:00:06 /opt/40/jdk1.8.0_161/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.xljvX8Ss -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/opt/40/elasticsearch-6.2.0 -Des.path.conf=/opt/40/elasticsearch-6.2.0/config -cp /opt/40/elasticsearch-6.2.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
justin 13366 13282 0 08:34 pts/2 00:00:00 grep --color=auto -i elas
[justin@localhost bin]$

發現啓動一會就自動關閉了。啓動時候不要後臺啓動，這樣能夠打印出日誌，方便找問題，等穩定了在後臺啓動

[justin@localhost bin]$ ./elasticsearch
[2018-02-05T11:39:47,616][INFO ][o.e.b.BootstrapChecks ] [IAWTuJo] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [3] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [3856] for user [justin] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-02-05T11:39:47,643][INFO ][o.e.n.Node ] [IAWTuJo] stopping ...
[2018-02-05T11:39:47,687][INFO ][o.e.n.Node ] [IAWTuJo] stopped
[2018-02-05T11:39:47,687][INFO ][o.e.n.Node ] [IAWTuJo] closing ...
[2018-02-05T11:39:47,723][INFO ][o.e.n.Node ] [IAWTuJo] closed
[justin@localhost bin]$

切換到root用戶，作以下修改

[root@localhost ~]# vim /etc/sysctl.conf
vm.max_map_count = 1048576
[root@localhost ~]# sysctl -p
[root@localhost ~]# vim /etc/security/limits.conf

• hard nofile 65536
• soft nofile 65536
  [root@localhost ~]# vim /etc/security/limits.d/20-nproc.conf
• soft nproc 65535
  root soft nproc unlimited
  [root@localhost ~]#

  再次啓動

  [root@localhost ~]# su - justin
  [ES@localhost ~]$ cd /opt/elasticsearch-6.2.0/bin/
  ERROR: [1] bootstrap checks failed
  [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
  [ES@localhost bin]$

Centos6不支持SecComp，而ES5.2.0後的版本默認bootstrap.system_call_filter爲true，在elasticsearch.yml中增長bootstrap.system_call_filter爲false，注意要在Memory下面: 
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false
看到如下界面說明啓動成功
![](https://s1.51cto.com/images/blog/201802/09/a7f945f39f7381f3096338d3b8805d67.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=)

-----
Elasticsearch Head是集羣管理、數據可視化、增刪改查、查詢語句可視化工具，它的安裝方式有兩種：命令安裝、包安裝安裝。也能夠從下面地址下載https://github.com/mobz/elasticsearch-head、
https://github.com/mobz/elasticsearch-head/archive/master.zip

[root@localhost opt]# git clone git://github.com/mobz/elasticsearch-head.git
Initialized empty Git repository in /opt/elasticsearch-head/.git/
remote: Counting objects: 4224, done.
remote: Total 4224 (delta 0), reused 0 (delta 0), pack-reused 4224
Receiving objects: 100% (4224/4224), 2.16 MiB | 46 KiB/s, done.
Resolving deltas: 100% (2329/2329), done.
[root@localhost opt]# wget https://nodejs.org/dist/v6.10.2/node-v6.10.2-linux-x64.tar.xz
[root@localhost opt]# vim /etc/profile
export PATH=$PATH::/opt/node-v8.9.4-linux-x64/bin
[root@localhost opt]# source !$
[root@localhost opt]# cd elasticsearch-head/
[root@localhost elasticsearch-head]# npm install
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! phantomjs-prebuilt@2.1.16 install: node install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the phantomjs-prebuilt@2.1.16 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2018-02-09T07_03_39_766Z-debug.log
[root@localhost elasticsearch-head]# npm install phantomjs-prebuilt@2.1.14 --ignore-scripts
[root@localhost elasticsearch-head]# npm install
[root@localhost elasticsearch-head]# npm run start

或者經過grunt來啓動

[root@localhost elasticsearch-head]# npm install -g grunt-cli
[root@localhost elasticsearch-head]# grunt server

修改elasticsearch的參數
            增長新的參數，這樣head插件能夠訪問es

[ES@localhost bin]$ pwd
/opt/elasticsearch-6.2.0/bin
[ES@localhost bin]$ vim ../config/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*" #前引號前，後引號後有空格
[ES@localhost bin]$

而後經過http://10.10.2.83:9100/訪問
或者直接用命令安裝

[root@localhost opt]# cd /opt/elasticsearch-6.2.1/bin
[root@localhost bin]# ./plugin install mobz/elasticsearch-head

而後能夠經過http://127.0.0.1:9200/_plugin/head/訪問

修改head插件源碼
            修改服務器監聽地址:Gruntfile.js

[root@localhost elasticsearch-head]# vim Gruntfile.js
connect: {
server: {
options: {
hostname: '0.0.0.0', #增長hostname屬性
port: 9100,
base: '.',
keepalive: true
}
}
}

修改鏈接地址：_site/app.js

[root@localhost elasticsearch-head]# vim _site/app.js
app.App = ui.AbstractWidget.extend({
defaults: {
base_uri: null
},
init: function(parent) {
this._super();
this.prefs = services.Preferences.instance();
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://10.10.2.83:9200"; #把localhost修改爲ES的服務器地址
if( this.base_uri.charAt( this.base_uri.length - 1 ) !== "/" ) {
// XHR request fails if the URL is not ending with a "/"
this.base_uri += "/";
}
if( this.config.auth_user ) {
var credentials = window.btoa( this.config.auth_user + ":" + this.config.auth_password );
$.ajaxSetup({
headers: {
"Authorization": "Basic " + credentials
}
});
}
[root@localhost elasticsearch-head]# nohup grunt server &

-----
x-pack安全模塊(security機制)
只要安裝好elasticsearch後，輸入正確的ip地址和端口，並有head插件就能夠看到索引庫，因此若是別人知道IP和和端口，那麼數據就徹底裸奔在別人面前。安裝x-pack後用戶名和密碼默認爲elastic和changeme。
官方文檔參考地址:https://www.elastic.co/guide/en/x-pack/5.4/logstash.html
一、修改Elasticsearch配置文件

[root@localhost bin]# pwd
/opt/elasticsearch-6.2.0/bin
[root@localhost bin]# ./elasticsearch-plugin install x-pack
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]ybr/>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin forks a native controller @br/>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.

Continue with installation? [y/N]y
Elasticsearch keystore is required by plugin [x-pack-security], creating...
-> Installed x-pack with: x-pack-security,x-pack-logstash,x-pack-deprecation,x-pack-watcher,x-pack-upgrade,x-pack-core,x-pack-ml,x-pack-monitoring,x-pack-graph
[root@localhost bin]#
[root@localhost elasticsearch-head]# vim /opt/elasticsearch-6.2.0/config/elasticsearch.yml
http.cors.allow-headers: "Authorization" #前引號前，後引號後有空格
[ES@localhost bin]$ ./elasticsearch -d

二、頁面訪問
http://10.10.2.83:9100/?auth_user=elastic&auth_password=changeme

x-pack也能夠建立一個權限組，而後建立用戶，給這個用戶分配權限。
1.設置權限組

http://10.10.2.83:9200/_xpack/security/role/logstash_writer/
{
 "cluster": ["manage_index_templates","monitor"],
 "indices": [
    {
     "names": [ "logstash-*","indexdb*" ],
     "privileges":["write","delete","create_index"]
    }
  ]
}

2.給用戶受權,這裏的logstash_internal爲用戶名。
http://10.10.2.83:9200/_xpack/security/user/logstash_internal
{
 "password" : "changeme",
 "roles" : [ "logstash_writer"],
 "full_name" : "Internal Logstash User"
}

若是不須要該插件，也能夠經過很是簡單的方式刪除

[root@localhost bin]# pwd
/opt/elasticsearch-6.2.0/bin
[root@localhost bin]# ./elasticsearch-plugin remove x-pack
-> removing [x-pack]...
-> preserving plugin config files [/opt/elasticsearch-6.2.0/config/x-pack] in case of upgrade; use --purge if not needed
[root@localhost bin]#

---

• 內存配置修改

配置 Elasticsearch 堆大小時須要經過 $ES_HEAP_SIZE 環境變量應用兩個規則：
不要超過可用 RAM 的 50%
Lucene 能很好利用文件系統的緩存，它是經過系統內核管理的。若是沒有足夠的文件系統緩存空間，性能會受到影響。 此外，專用於堆的內存越多意味着其餘全部使用 doc values 的字段內存越少。
不要超過 32 GB
若是堆大小小於 32 GB，JVM 能夠利用指針壓縮，這能夠大大下降內存的使用：每一個指針 4 字節而不是 8 字節。

[justin@es-2-10-0005 bin]$ vim ../config/jvm.options
#Xms represents the initial size of total heap space
#Xmx represents the maximum size of total heap space

-Xms16g
-Xmx16g

################################################################
#Expert settings

OK