CentOS 7 搭建ELK

時間 2019-12-19

標籤 centos 搭建 elk 欄目 CentOS 简体版

原文原文鏈接

#</center> CentOS 7 搭建ELKjava

一、基礎環境配置

操做系統及各組件版本
centos-7-x86_64
java1.8
elasticsearch-6.2.4
kibana-6.2.4
logstash-6.2.4

1.配置java 環境變量，我這裏使用的是java8

export JAVA_HOME=/usr/java/jdk1.8.0_144
export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin

2.查看java 版本以及安裝是否正確

[root@lastsummer130 java]# source  /etc/profile
[root@lastsummer130 java]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

3.增長elasticsearch用戶

[root@lastsummer130 ~]# useradd -m -d /home/elasticsearch elasticsearch
[root@lastsummer130 ~]# passwd  elasticsearch 
Changing password for user elasticsearch.
New password: 
BAD PASSWORD: The password fails the dictionary check - it is too simplistic/systematic
Retype new password: 
passwd: all authentication tokens updated successfully.

4.配置visudo 權限

[root@lastsummer130 ~]# visudo 
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL
elasticsearch ALL=(ALL)       ALL
## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL
elasticsearch   ALL=(ALL)       NOPASSWD: ALL

5.修改系統參數

[elasticsearch@lastsummer130]$ sudo vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

[elasticsearch@lastsummer130]$ sudo vi /etc/sysctl.conf
vm.max_map_count=655360
[elasticsearch@lastsummer130]$ sudo sysctl -p

2、elasticsearch

1.導入祕鑰

[root@lastsummer130 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2.安裝elasticsearch

[root@lastsummer130 tools]# rpm -ivh elasticsearch-6.2.4.rpm
Preparing...                          ################################# [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Updating / installing...
   1:elasticsearch-0:6.2.4-1          ################################# [100%]
Job for systemd-sysctl.service failed because the control process exited with error code. See "systemctl status systemd-sysctl.service" and "journalctl -xe" for details.
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

3.修改文件權限

[elasticsearch@lastsummer130 ~]$ sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch/
[elasticsearch@lastsummer130 ~]$ sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
[elasticsearch@lastsummer130 ~]$ sudo chown -R elasticsearch:elasticsearch /usr/lib/systemd/system/elasticsearch.service
[elasticsearch@lastsummer130 ~]# chown -R elasticsearch:elasticsearch /var/run/elasticsearch/
[elasticsearch@lastsummer130 ~]# chown -R elasticsearch:elasticsearch /etc/sysconfig/elasticsearch

4.修改文件elasticsearch.service中的用戶組和用戶

/usr/lib/systemd/system/elasticsearch.service
User=elasticsearch
Group=elasticsearch

5.裝置服務

[elasticsearch@lastsummer130]$ sudo systemctl daemon-reload
[elasticsearch@lastsummer130]$ sudo systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

6.修改配置文件：記得修改data和log文件的權限

[root@lastsummer130 ~]# vim /etc/elasticsearch/elasticsearch.yml
#數據存放目錄
path.data: /var/lib/elasticsearch
#日誌存放目錄
path.logs: /var/log/elasticsearch
#本身的ip
network.host: 192.168.145.130
#訪問端口
http.port: 9200

[root@lastsummer130 ~]# chown  elasticsearch.elasticsearch  /var/log/elasticsearch/ /var/lib/elasticsearch/

7.啓動

[elasticsearch@lastsummer130]$ sudo systemctl start elasticsearch.service
[root@lastsummer130 java]# sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-08-26 17:52:56 CST; 15s ago
     Docs: http://www.elastic.co
  Process: 24538 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 24538 (code=exited, status=1/FAILURE)

Aug 26 17:52:56 lastsummer130.com systemd[1]: Started Elasticsearch.
Aug 26 17:52:56 lastsummer130.com elasticsearch[24538]: which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
Aug 26 17:52:56 lastsummer130.com systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Aug 26 17:52:56 lastsummer130.com systemd[1]: Unit elasticsearch.service entered failed state.
Aug 26 17:52:56 lastsummer130.com systemd[1]: elasticsearch.service failed.

8.查看緣由

根據日誌進行分析是elasticsearch 沒有找到java 程序致使啓動失敗

9.修改配置

[elasticsearch@lastsummer130 ~]$ vim /etc/sysconfig/elasticsearch 
# Elasticsearch Java path 配置java home
JAVA_HOME=/usr/java/jdk1.8.0_144

10.啓動測試

[elasticsearch@lastsummer130 ~]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2019-08-26 17:57:44 CST; 7s ago
     Docs: http://www.elastic.co
 Main PID: 29678 (java)
   CGroup: /system.slice/elasticsearch.service
           └─29678 /usr/java/jdk1.8.0_144/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt...
8月 26 17:57:44 lastsummer130.com systemd[1]: Started Elasticsearch.
啓動成功

11.常見報錯

[2019-08-27T09:10:05,101][ERROR][o.e.b.Bootstrap          ] [j_wn2C2] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked

vim /etc/security/limits.conf //添加, 【註銷後並從新登陸生效】
* soft nofile 300000
* hard nofile 300000
* soft nproc 102400
* hard nproc 102400

vim /etc/security/limits.conf //添加
* soft memlock unlimited
* hard memlock unlimited
驗證是否生效
ulimit -a

3、kibana

1.安裝kibana

[root@lastsummer130 tools]# rpm -ivh kibana-6.2.4-x86_64.rpm 
warning: kibana-6.2.4-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:kibana-6.2.4-1                   ################################# [100%]

2.配置kibana.yml

[root@lastsummer130 tools]# vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"

3.啓動kibana

[root@lastsummer130 tools]# systemctl enable kibana
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
[root@lastsummer130 tools]# systemctl start kibana
[root@lastsummer130 tools]# netstat -plntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1319/master         
tcp        0      0 192.168.145.130:5601    0.0.0.0:*               LISTEN      59983/node          
tcp        0      0 0.0.0.0:2223            0.0.0.0:*               LISTEN      1026/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1319/master         
tcp6       0      0 :::2223                 :::*                    LISTEN      1026/sshd           
tcp6       0      0 192.168.145.130:9200    :::*                    LISTEN      1022/java           
tcp6       0      0 192.168.145.130:9300    :::*                    LISTEN      1022/java           
udp        0      0 127.0.0.1:323           0.0.0.0:*                           770/chronyd         
udp6       0      0 ::1:323                 :::*                                770/chronyd

和elasticsearch同樣，最後經過netstat -plntu查看kibana是否啓動成功，若是有端口號爲5601的輸出那就表明kibana啓動成功了node

4、logstash

1.安裝logstash

[root@lastsummer130 tools]# rpm -ivh logstash-6.2.4.rpm 
warning: logstash-6.2.4.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.2.4-1               ################################# [100%]
which: no java in (/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin)
could not find java; set JAVA_HOME or ensure java is in PATH
chmod: cannot access ‘/etc/default/logstash’: No such file or directory

2.根據提示排錯

提示是沒有找到java 環境，我這裏的java 環境是shell

[root@lastsummer130 tools]# which java
/usr/java/jdk1.8.0_144/bin/java
建立軟鏈接
[root@lastsummer130 tools]# ln -s /usr/java/jdk1.8.0_144/bin/java /usr/bin/java

3.從新進行安裝logstash

[root@lastsummer130 tools]# rpm -ivh logstash-6.2.4.rpm 
warning: logstash-6.2.4.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:6.2.4-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash

4.啓動並檢查狀態

[root@lastsummer130 tools]# systemctl enable logstash
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[root@lastsummer130 tools]# systemctl start logstash
[root@lastsummer130 tools]# ps -ef | grep logstash
logstash  87046      1 93 15:15 ?        00:00:10 /bin/java -Xms256m -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -cp /usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-19.0.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
root      87270   1604  0 15:16 pts/0    00:00:00 grep --color=auto logstash