ELK Stack 5.2.2 安裝文檔

簡介：

ELK Stack 安裝文檔，此次都使用最新版本(5.2.2)、RPM 包的方式搭建 ELK Stack。

下載地址:

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.rpm
https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.rpm
https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-x86_64.rpm

jre: http://javadl.oracle.com/webapps/download/AutoDL?BundleId=216423
jdk: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

# 若是隻是須要 java 環境，那麼安裝 jre 便可，如還需編譯 java 包就須要安裝 jdk 了。
# 運行 Elasticsearch jre 便可

1、安裝

shell > rpm -e elasticsearch
Stopping elasticsearch service... OK
warning: /etc/sysconfig/elasticsearch saved as /etc/sysconfig/elasticsearch.rpmsave
warning: /etc/init.d/elasticsearch saved as /etc/init.d/elasticsearch.rpmsave
warning: /etc/elasticsearch/elasticsearch.yml saved as /etc/elasticsearch/elasticsearch.yml.rpmsave
Deleting log directory... OK
Deleting plugins directory... OK

shell > rpm -e kibana
Stopping kibana service... OK
warning: /opt/kibana/config/kibana.yml saved as /opt/kibana/config/kibana.yml.rpmsave

# 我以前使用 2.4.1 版本，也是經過 rpm 安裝的，須要先卸載。

shell > rm -rf /etc/sysconfig/elasticsearch.rpmsave 
shell > rm -rf /etc/init.d/elasticsearch.rpmsave 
shell > rm -rf /etc/elasticsearch/elasticsearch.yml.rpmsave 
shell > rm -rf /opt/kibana/config/kibana.yml.rpmsave

# 能夠看到卸載的時候，這些文件沒有被刪除，強迫症的我是不容許這些文件存在的。

shell > cd /usr/local/src; rpm -ivh elasticsearch-5.2.2.rpm logstash-5.2.2.rpm kibana-5.2.2-x86_64.rpm

# 因爲是測試一下新版本，因此都裝在了一臺服務器上。

2、配置

一、Elasticsearch

shell > grep -vP '^#|^$' /etc/elasticsearch/elasticsearch.yml 
# 集羣名稱
cluster.name: elk
# 節點名稱
node.name: node-1
# 數據路徑
path.data: /data/elast/data
# 日誌路徑
path.logs: /data/elast/logs
# 啓動時鎖住內存，防止數據被交換到 SWAP
bootstrap.memory_lock: true
# 監聽地址
network.host: 0.0.0.0
# 與其他節點通訊地址
network.publish_host: 10.127.174.217
# 開啓 HTTP 協議
http.port: 9200
# 解決啓動報錯
bootstrap.system_call_filter: false

shell > mkdir -p /data/elast/{data,logs}
shell > chown -R elasticsearch.elasticsearch /data/elast

# 建立數據、日誌目錄

二、Logstash

shell > vim /etc/logstash/conf.d/for_elk.conf
# 輸入插件，這裏從 redis 中讀取數據
input {
    redis {
        data_type => "list"
        key => "for_elk"
        host => "10.217.79.61"
        port => 6379
        threads => 10
    }
}
# 過濾插件，按需切割日誌、加減字段等
filter {
    mutate {
        split => ["message", "|"]
        add_field => {"clientip" => "%{message[0]}"}
        add_field => {"localtime" => "%{message[1]}"}
        add_field => {"api" => "%{message[2]}"}
        add_field => {"request_all" => "%{message[3]}"}
        add_field => {"http_code" => "%{message[4]}"}
        add_field => {"request_body" => "%{message[6]}"}
        add_field => {"request_time" => "%{message[11]}"}
    }

    date {
        match => ["localtime", "dd/MMM/yyyy:HH:mm:ss Z"]
    }

    geoip {
        source => "clientip"
        fields => ["city_name", "latitude", "longitude"]
    }

    kv {
        source => "request_body"
        field_split => "&"
        remove_field => "host"
        remove_field => "path"
        remove_field => "message"
        remove_field => "request_all"
        remove_field => "request_body"
    }

    mutate {
        convert => [
            "id", "integer",
            "cid", "integer",
            "tid", "integer",
            "vid", "integer",
            "version", "float",
            "http_code", "integer",
            "request_time", "float"
        ]
    }
}
# 輸出插件
output {
    elasticsearch { 
        hosts => ["10.127.174.217:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        template_overwrite => true
    } 
    # stdout {
    #     codec => rubydebug
    # }
}

# 能夠測試可否從 redis 拿到數據，而後在作 filter，最後測試可否寫入 elasticsearch

三、Kibana

shell > /etc/kibana/kibana.yml

# Kibana 其實不用修改，暫時採用默認配置便可

3、啓動

一、Elasticsearch

shell > /etc/init.d/elasticsearch start

二、Logstash

shell > /usr/share/logstash/bin/logstash --path.settings /etc/logstash > /dev/null &

三、Kibana

shell > /etc/init.d/kibana start

4、訪問

# http://x.x.x.x:5601 便可，根據 index 創建索引，嗯 確實比 K4 漂亮

5、插件安裝

一、Elasticsearch head (從 5.0 起，該插件以一個單獨的服務運行)

shell > cd /usr/local

shell > git clone git://github.com/mobz/elasticsearch-head.git

shell > cd elasticsearch-head

shell > npm install

shell > vim Gruntfile.js

                connect: {
                        server: {
                                options: {
                                        hostname: '0.0.0.0',
                                        port: 9100,
                                        base: '.',
                                        keepalive: true
                                }
                        }
                }

# 默認只監聽 127.0.0.1，因此要加上 hostname: '0.0.0.0'

shell > ./node_modules/grunt/bin/grunt server > /dev/null &

shell > vim /etc/elasticsearch/elasticsearch.yml

# head plugin
http.cors.enabled: true
http.cors.allow-origin: "*"

# elasticsearch 5.x 須要設置該參數，不然沒法 head 沒法鏈接 es
# 你可能注意到 es 集羣狀態爲 yellow，不要慌...
# 那是由於副本不可用，由於只有一個 es 節點，而副本不能在本機，不礙事 ！

二、IK Analysis for Elasticsearch

shell > wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.tar.gz
shell > tar zxf apache-maven-3.5.0-bin.tar.gz -C /usr/local
shell > echo -e '\nexport JAVA_HOME=/usr/java/default' >> /etc/profile && source /etc/profile

shell > wget https://github.com/medcl/elasticsearch-analysis-ik/archive/v5.2.2.zip
shell > unzip v5.2.2.zip
shell > cd elasticsearch-analysis-ik-5.2.2
shell > /usr/local/apache-maven-3.5.0/bin/mvn package
shell > unzip target/releases/elasticsearch-analysis-ik-5.2.2.zip -d /usr/share/elasticsearch/plugins/ik
shell > /usr/share/elasticsearch/bin/elasticsearch-plugin list
ik
shell > /etc/init.d/elasticsearch restart

附件：

一、Elasticsearch 啓動報錯

> bootstrap.memory_lock: true 參數致使

memory locking requested for elasticsearch process but memory is not locked

解決方法:

shell > vim /etc/security/limits.conf

# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

> CentOS 6.x 不支持 CONFIG_SECCOMP 致使

[2017-03-01T12:00:53,986][WARN ][o.e.b.JNANatives         ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
        at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:215) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:99) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:203) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.2.2.jar:5.2.2]

bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

解決方法：

shell > vim /etc/elasticsearch/elasticsearch.yml

bootstrap.system_call_filter: falses

> /etc/security/limits.d/90-nproc.conf 默認參數太低致使啓動失敗

[2017-07-06T14:57:47,840][ERROR][o.e.b.Bootstrap          ] [node01] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

解決方法：

shell > vim /etc/security/limits.d/90-nproc.conf

*          soft    nproc     2048
root       soft    nproc     unlimited

# 將原 1024 改成 2048