elasticsearch+logstash+kibana部署

這篇博客講的是elasticsearch+logstash+kibana部署的方法。

內容大綱：

一、elasticsearch+logstash+kibana部署

二、收集Tomcat日誌

三、收集tcp日誌

1、elasticsearch+logstash+kibana部署

一、elasticsearch部署

因爲條件有限，這三個組件都裝在同一臺機器上面。

系統版本：

elasticsearch-6.7.0、logstash-6.7.0、kibana-6.7.0-linux-x86_64

[root@openstack ~]# cd /usr/local/elasticsearch-6.7.0/
[root@openstack elasticsearch-6.7.0]# ls
bin  config  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.textile
[root@openstack elasticsearch-6.7.0]# vim config/elasticsearch.yml

[root@openstack elasticsearch-6.7.0]# grep '^[a-z]' config/elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /elk/data
path.logs: /elk/logs
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"

[root@openstack ~]# mkdir -p /elk/data/ /elk/logs/
[root@openstack ~]# useradd elasticsearch
[root@openstack ~]# chown -R elasticsearch:elasticsearch /elk/
[root@openstack ~]# chown -R elasticsearch:elasticsearch /usr/local/elasticsearch-6.0.0
[root@openstack ~]# su - elasticsearch
[elasticsearch@openstack ~]$ cd /usr/local/elasticsearch-6.0.0/
[elasticsearch@openstack bin]$ ./elasticsearch -d           後臺啓動，此時啓動不了，須要修改一下配置

[root@openstack ~]# vim /etc/security/limits.conf    重啓參數才能生效

*        hard    nofile           65536
*        soft    nofile           65536
*        soft    memlock          unlimited
*        hard    memlock          unlimited

[root@openstack ~]# vim /etc/sysctl.conf
vm.max_map_count=262144
[root@openstack ~]# sysctl -p    參數生效

切換用戶以後從新啓動，查看進程和監聽端口
[root@openstack ~]# ps -ef|grep java                               查看進程
elastic+   6750      1  1 12:29 pts/2    00:02:30 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/local/elasticsearch-6.0.0 -Des.path.conf=/usr/local/elasticsearch-6.0.0/config -cp /usr/local/elasticsearch-6.0.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d

[root@openstack ~]# netstat -tlunp|grep java                   查看端口
tcp6       0      0 :::9200                 :::*                    LISTEN      6750/java           
tcp6       0      0 :::9300                 :::*                    LISTEN      6750/java

[root@openstack ~]# curl 192.168.209.7:9200       查看信息
{
  "name" : "node-1",
  "cluster_name" : "my-application",
  "cluster_uuid" : "Z3TOUu0jTUiCSHsuFJsv_g",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "8f0685b",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

二、kibana部署：

[root@openstack ~]# cd /usr/local/src/kibana-6.7.0-linux-x86_64/
[root@openstack kibana-6.7.0-linux-x86_64]# grep '^[a-z]' config/kibana.yml
server.port: 5601
server.host: "192.168.199.7"
elasticsearch.url: "http://192.168.199.7:9200"
kibana.index: ".kibana"
[root@openstack ~]# useradd kibana
[root@openstack ~]# chown -R kibana:kibana /usr/local/src/kibana-6.7.0-linux-x86_64

[root@openstack ~]# su - kibana
[kibana@openstack ~]$ cd /usr/local/src/kibana-6.7.0-linux-x86_64/bin/
[kibana@openstack bin]$ nohup ./kibana &
[root@openstack ~]# ps -ef|grep kibana
kibana     3704      1  2 12:20 pts/1    00:08:52 ./../node/bin/node --no-warnings --max-http-header-size=65536 ./../src/cli
[root@openstack ~]# netstat -tlunp|grep 5601
tcp        0      0 192.168.199.7:5601      0.0.0.0:*               LISTEN      3704/./../node/bin/

在頁面訪問：

kibana部署完成。

三、logstash部署

[root@openstack ~]# cd /usr/local/src/logstash-6.7.0/
本身寫一下配置文件

[root@openstack logstash-6.7.0]# cat config/tomcat.conf
input {
     file {
        path => ["/app/tomcat/logs/*"]
    type => "tomcat-access-log-5612"
        start_position => "beginning"
    stat_interval => "2"
    codec => "json"
     }
}


output {
       elasticsearch {
          hosts => ["192.168.199.7:9200"]
          index => "logstash-tomcat-access-log-5612-%{+YYYY.MM.dd}"
       }
    file {
      path => "/tmp/tomcat.txt"
    }
}

[root@openstack logstash-6.7.0]# ./bin/logstash -f config/tomcat.conf -t    測試一下配置文件是否正確
Sending Logstash logs to /usr/local/src/logstash-6.7.0/logs which is now configured via log4j2.properties
[2019-04-05T19:04:13,134][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-04-05T19:04:37,134][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
有這個字段（Configuration OK）表示沒有問題，去掉-t以後就能夠啓動了。

logstash部署完成。

 

2、收集Tomcat日誌。

簡單安裝一下Tomcat

[root@openstack ~]# cd /app/apache-tomcat-8.5.35/
[root@openstack ~]# cd /app/
[root@openstack app]# ln -sv /app/apache-tomcat-8.5.35 /app/tomcat
[root@openstack app]# ll
總用量 0
drwxr-xr-x. 9 root root 220 4月   5 12:40 apache-tomcat-8.5.35
lrwxrwxrwx. 1 root root  25 4月   5 12:41 tomcat -> /app/apache-tomcat-8.5.35
[root@openstack app]# cd tomcat/webapps/
[root@openstack webapps]# rm -rf *
[root@openstack webapps]# mkdir test
[root@openstack test]# cat index.html
tomcat page
[root@openstack bin]# cd /app/tomcat/bin/
[root@openstack bin]# ./startup.sh
[root@openstack bin]# netstat -tlunp|grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      10106/java
頁面訪問：

 

 

啓動一下logstash讓日誌傳過來，而後在頁面查看：

此時就有了日誌。

 

3、收集tcp日誌

一、測試

[root@openstack ~]# cd /usr/local/src/logstash-6.7.0/
[root@openstack logstash-6.7.0]# cat config/tcp.conf
input {
   tcp {
     port => "8443"
     type => "tcplog"
    }
}

output {
   stdout {
     codec => "rubydebug"    
    }
}

[root@openstack logstash-6.7.0]# ./bin/logstash -f config/tcp.conf.bak -t
Sending Logstash logs to /usr/local/src/logstash-6.7.0/logs which is now configured via log4j2.properties
[2019-04-05T19:30:58,968][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-04-05T19:31:09,882][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

 

[root@openstack ~]# yum -y install nc

查看一下，logstash的配置文件名稱能夠隨便命名，只要指定就行

測試沒有問題，此時須要把日誌打到elasticsearch中。

修改一下配置文件：

[root@openstack logstash-6.7.0]# cat config/tcp.conf
input {
   tcp {
     port => "8443"
     type => "tcplog-5612"
    }
}

output {
   elasticsearch {
     hosts => ["192.168.199.7:9200"]
     index => "tcplog-5612-%{+YYYY.MM.dd}"
   }
    
}啓動logstash服務

在kibana頁面配置：

 

 再次建立index。此時有以前建立的index，也有日誌了。

 

 這樣就OK了！！！！！！！！！！！

ES還有更增強大的功能，和諸位一塊兒研究，學習。