ELK5.2+kafka+zookeeper+filebeat集羣部署

時間 2019-11-11

標籤 elk5.2+kafka+zookeeper+filebeat elk kafka zookeeper filebeat 集羣部署欄目 Kafka 简体版

原文原文鏈接

架構圖java

考慮到日誌系統的可擴展性以及目前的資源（部分功能複用），整個ELK架構以下：

架構解讀 : （整個架構從左到右，總共分爲5層）node

第一層、數據採集層python

最左邊的是業務服務器集羣，上面安裝了filebeat作日誌採集，同時把採集的日誌分別發送給兩個logstash服務（2.18七、2.189）mysql

第二層、數據處理層，數據緩存層linux

logstash服務把接受到的日誌通過格式處理，轉存到本地的kafka broker+zookeeper 集羣中。nginx

第三層、數據轉發層git

這個單獨的Logstash（2.184）節點會實時去kafka broker集羣拉數據，轉發至ES DataNode。github

第四層、數據持久化存儲sql

ES DataNode 會把收到的數據，寫磁盤，建索引庫。apache

第五層、數據檢索，數據展現

ES Master + Kibana 主要協調 ES集羣，處理數據檢索請求，數據展現。

服務器資源以及軟件版本
- 操做系統：centos7.二、虛擬機

服務器角色（研究環境）

主機ip	部署服務	服務器配置
192.168.2.184	elastic、kafka、 logstash-out-from-kafka zookeeper
192.168.2.187	elastic、kafka、logstash-in-to-kafka、zookeeper
192.168.2.189	elastic、kafka、logstash-in-to-kafka、zookeeper、kibana

羣星日誌系統服務器角色（測試環境）

主機ip	部署服務	服務器配置
192.168.2.130	ExceptionLess
192.168.2.131	kafka、 logstash-out-from-kafka zookeeper	opskafka.manjinba.cn
192.168.2.132	kafka、logstash-in-to-kafka、zookeeper	opskafka.manjinba.cn
192.168.2.133	kafka、logstash-in-to-kafka、zookeeper	opskafka.manjinba.cn
192.168.2.135	elastic	opselastic.manjinba.cn
192.168.2.136	elastic	opselastic.manjinba.cn
192.168.2.138	kibana	kibana.manjinba.cn

軟件版本：
jdk-8u151-linux-x64
elasticsearch-5.2.2 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.2.tar.gz

kafka_2.10-0.10.2.0 wget http://mirror.bit.edu.cn/apache/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz
kafka-manager wget https://github.com/yahoo/kafka-manager/archive/master.zip
kibana-5.2.2-linux-x86_64 wget https://artifacts.elastic.co/downloads/kibana/kibana-5.2.2-linux-x86_64.tar.gz
logstash-5.2.2 wget https://artifacts.elastic.co/downloads/logstash/logstash-5.2.2.tar.gz
zookeeper-3.4.9 wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

filebeat-5.2.2 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.2.2-linux-x86_64.tar.gz

安裝部署

系統優化

 
            cat /etc/sysctl.conf 
           
            net.ipv4.tcp_max_syn_backlog = 4096 
           
            net.core.netdev_max_backlog = 2048 
           
            net.ipv4.tcp_fin_timeout = 15 
           
            net.ipv4.tcp_tw_reuse = 1 
           
            net.ipv4.tcp_tw_recycle = 1 
           
            net.ipv4.tcp_syncookies = 1 
           
            vm.max_map_count= 262144 #後期配置ES很關鍵 
           
            vm.swappiness = 1

 
            cat /etc/security/limits.conf 
           
            *                               soft    nofile  65536 
           
            *                               hard    nofile  65536

配置java環境

 
            cd /apps/svr 
           
            tar zxvf jdk-8u151-linux-x64.tar.gz 
           
            ln -s jdk1.8.0_151 jdk 
           
            cat >> /etc/profile <<EOF 
           
            export JAVA_HOME=/apps/svr/jdk 
           
            export PATH=$JAVA_HOME/bin:$PATH 
           
            export CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar 
           
            EOF 
           
            source /etc/profile

用戶問題
爲了方便這裏全部的應用所有都在apps賬號下
useradd apps && echo "Qwer1234" | passwd --stdin apps

python升級以及安裝supervisor

 
            cat update_python.sh 
           
            #!/bin/bash 
           
            #creat by xiaojs 
           
            if  
            [ whoami !=  
            'root'  
            ] 
           
            then 
           
            exit 1 
           
            fi 
           
            if  
            [[ python -c  
            "import platform ;print platform.python_version()"  
            = 2.7.* ]] 
           
            then 
           
            echo  
            'you need not do everything' 
           
            exit 0 
           
            else 
           
            echo  
            '============================' 
           
            echo  
            '=======start update========' 
           
            fi 
           
            # get the tar 
           
            cd /usr/local/src 
           
            wget http: 
            //ops.bubugao-inc.com/python/Python-2.7.8.tgz 
           
            wget http: 
            //ops.bubugao-inc.com/python/pyinotify.tar.gz 
           
            wget http: 
            //ops.bubugao-inc.com/python/MySQL-python-1.2.4.zip 
           
            ## 
           
            yum -y install git gcc mysql mysql-devel 
           
            #install 
           
            tar zxvf Python-2.7.8.tgz 
           
            cd Python-2.7.8 
           
            ./configure --prefix=/usr/local/python2.7.8 
           
            make && make install 
           
            mv /usr/bin/python /usr/bin/python_old 
           
            ln -s /usr/local/python2.7.8/bin/python /usr/bin/ 
           
            sed -i  
            's/python/python_old/1'  
            /usr/bin/yum 
           
            #intall the plugin 
           
            cd .. 
           
            tar zxvf pyinotify.tar.gz 
           
            cd pyinotify 
           
            python setup.py install 
           
            cd .. 
           
            unzip MySQL-python-1.2.4.zip 
           
            cd MySQL-python-1.2.4 
           
            python setup.py install 
           
            ####install supervisor 
           
            cd /usr/local/src 
           
            wget --no-check-certificate https: 
            //bootstrap.pypa.io/ez_setup.py -O - | sudo python 
           
            wget http: 
            //pypi.python.org/packages/source/d/distribute/distribute-0.6.10.tar.gz 
           
            tar xf distribute-0.6.10.tar.gz 
           
            cd distribute-0.6.10 
           
            python setup.py install 
           
            easy_install supervisor 
           
            cd /usr/local/python2.7.8/bin/ 
           
            cp supervisord supervisorctl echo_supervisord_conf /usr/bin/ 
           
            mkdir /etc/supervisor && cd /etc/supervisor 
           
            wget http: 
            //ops.bubugao-inc.com/python/supervisord.conf

安裝elasticsearch
cd /apps/svr/
tar zxvf elasticsearch-5.2.2.tar.gz
ln -s elasticsearch-5.2.2 elasticsearch
[root@17161 elasticsearch]# sed -n /^[^#]/p config/elasticsearch.yml

 
            cluster.name: SuperApp 
           
            node.name: manjinba01 
           
            network.host: 0.0.0.0 
           
            http.port: 9200 
           
            discovery.zen.ping.unicast.hosts: [ 
            "192.168.2.184:9300" 
            , 
            "192.168.2.187:9300" 
            , 
            "192.168.2.189:9300" 
            ] 
           
            discovery.zen.minimum_master_nodes: 1 
           
            bootstrap.system_call_filter:  
            false 
           
            bootstrap.memory_lock:  
            false 
           
            http.cors.enabled:  
            true 
           
            http.cors.allow-origin:  
            "*"

啓動elasticsearch

chown -R apps.apps /apps

su - apps

cd /apps/svr/elasticsearch

bin/elasticsearch -d

另外兩臺相似，後續會安裝x-pack，因此之前的head和bigdesk不用安裝
zookeeper+kafka集羣部署

#zookeeper
cd /apps/svr
tar zxvf zookeeper-3.4.9.tar.gz
ln -s zookeeper-3.4.9 zookeeper
mkdir -p /apps/dbdat/zookeeper
[root@17163 zookeeper]# sed -n ‘/^[^#]/p’ conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/apps/dbdat/zookeeper
clientPort=2181
server.1=192.168.2.184:12888:13888
server.2=192.168.2.187:12888:13888
server.3=192.168.2.189:12888:13888

#三臺服務器分別賦值
echo 1 > /apps/dbdat/zookeeper/myid
echo 2 > /apps/dbdat/zookeeper/myid
echo 3 > /apps/dbdat/zookeeper/myid

#啓動並查看狀態
/apps/svr/zookeeper/bin/zkServer.sh start
/apps/svr/zookeeper/bin/zkServer.sh status
[root@17163 zookeeper]# /apps/svr/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /apps/svr/zookeeper/bin/../conf/zoo.cfg
Mode: follower
#以上信息就是沒問題

#kafka集羣

 
           cd /apps/svr 
          
           tar zxvf kafka_2.10-0.10.2.0.tgz 
          
           ln -s kafka_2.10-0.10.2.0 kafka 
          
           [root@17161 src]# sed -n  
           '/^[^#]/p'  
           /apps/svr/kafka/config/server.properties 
          
           broker.id=1 
          
           delete.topic.enable= 
           true 
          
           listeners=PLAINTEXT: 
           //192.168.2.184:9092 
          
           num.network.threads=3 
          
           num.io.threads=8 
          
           socket.send.buffer.bytes=102400 
          
           socket.receive.buffer.bytes=102400 
          
           socket.request.max.bytes=104857600 
          
           log.dirs=/apps/logs/kafka-logs 
          
           num.partitions=1 
          
           num.recovery.threads.per.data.dir=1 
          
           log.retention.hours=168 
          
           log.segment.bytes=1073741824 
          
           log.retention.check.interval.ms=300000 
          
           zookeeper.connect=192.168.2.184:2181,192.168.2.187:2181,192.168.2.189:2181 
          
           zookeeper.connection.timeout.ms=6000

#不一樣的節點，注意broker.id和linsten的ip

 
           \#啓動查看是否正常 
          
           nohup /apps/svr/kafka/bin/kafka-server-start.sh /apps/svr/kafka/config/server.properties & 
          
           \#有一些用獲得的指令 
          
           bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test  # 建立topic 
          
           bin/kafka-topics.sh --list --zookeeper localhost:2181   # 查看已經建立的topic列表 
          
           bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test  # 查看topic的詳細信息 
          
           bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test # 發送消息, 回車後模擬輸入一下消息 
          
           bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test # 消費消息, 能夠換到其餘kafka節點, 同步接收生產節點發送的消息 
          
           bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic test --partitions 6  # 給topic增長分區 
          
           bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test1  # 刪除已經建立的topic, 前提是開了delete.topic.enable= 
           true 
           參數 
          
           若是還不能刪除, 能夠到zookeeper中去幹掉它 
          
           cd /usr/local/zookeeper-3.4.10/ 
          
           bin/zkCli.sh 
          
           ls /brokers/topics            # 查看topic 
          
           rm -rf /brokers/topics/test1     # 刪除topic

logstash的部署和配置

cd /apps/svr
tar zxvf logstash-5.2.2.tar.gz
ln -s logstash-5.2.2/ logstash
#安裝都同樣，重點是兩端配置文件不同，一個是負責寫入kafka，一個是負責從kafka提取出來寫入elasticsearch,配置分別以下：
[root@17162 ~]# cat /apps/conf/logstash/logstash-in-kafka.conf

 
           input { 
          
           beats { 
          
           port => 5044 
          
           } 
          
           } 
          
           output { 
          
           if  
           [type] ==  
           "nginx-accesslog"  
           { 
          
           kafka { 
          
           bootstrap_servers =>  
           "192.168.2.184:9092,192.168.2.187:9092,192.168.2.189:9092" 
          
           topic_id =>  
           "nginx-accesslog" 
          
           } 
          
           } 
          
           if  
           [type] ==  
           "tomcat-log"  
           { 
          
           kafka { 
          
           bootstrap_servers =>  
           "192.168.2.184:9092,192.168.2.187:9092,192.168.2.189:9092" 
          
           topic_id =>  
           "tomcat-log" 
          
           } 
          
           } 
          
           if  
           [type] ==  
           "sys-messages"  
           { 
          
           kafka { 
          
           bootstrap_servers =>  
           "192.168.2.184:9092,192.168.2.187:9092,192.168.2.189:9092" 
          
           topic_id =>  
           "sys-messages" 
          
           } 
          
           } 
          
           }

 
           [apps@17161 ~]$ cat /apps/conf/logstash/logstash-kafka.conf 
          
           input { 
          
           kafka{ 
          
           bootstrap_servers =>  
           "192.168.2.184:9092,192.168.2.187:9092,192.168.2.189:9092" 
          
           topics =>  
           "nginx-accesslog" 
          
           consumer_threads => 50 
          
           decorate_events =>  
           true 
          
           type =>  
           "nginx-accesslog" 
          
           } 
          
           kafka{ 
          
           bootstrap_servers =>  
           "192.168.2.184:9092,192.168.2.187:9092,192.168.2.189:9092" 
          
           topics =>  
           "sys-messages" 
          
           consumer_threads => 50 
          
           decorate_events =>  
           true 
          
           type =>  
           "sys-messages" 
          
           } 
          
           kafka{ 
          
           bootstrap_servers =>  
           "192.168.2.184:9092,192.168.2.187:9092,192.168.2.189:9092" 
          
           topics =>  
           "tomcat-log" 
          
           consumer_threads => 50 
          
           decorate_events =>  
           true 
          
           type =>  
           "tomcat-log" 
          
           } 
          
           } 
          
           filter { 
          
           if  
           [type] ==  
           "nginx-accesslog"  
           { 
          
           grok { 
          
           match => [ 
           "message" 
           , 
           "%{IPORHOST:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:rawrequest})\" (?:%{URIHOST:domain}|-) %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:x_forword} %{QS:upstream_host} %{QS:upstream_response} (%{WORD:upstream_cache_status}|-) %{QS:upstream_content_type} %{QS:upstream_response_time} > (%{BASE16FLOAT:request_time}) \"(%{NGINXUID:uid}|-)\"" 
           ] 
          
           } 
          
           date { 
          
           locale =>  
           "en_US" 
          
           match => [ 
           "timestamp" 
           ,  
           "dd/MMM/yyyy:HH:mm:ss Z" 
           ] 
          
           remove_field => [  
           "timestamp"  
           ] 
          
           } 
          
           } 
          
           if  
           [type] ==  
           "tomcat-log"  
           { 
          
           grok { 
          
           match => { 
           "message"  
           =>   
           "((app=(?<app>[^,]*)\,?))(\s*)((app0=(?<app0>[^,]*)\,?)?)(\s*)((app1=(?<app1>[^,]*)\,?)?)(.*\, host)(=(?<host>[^,]*)\,)(\s*)(pid=(?<pid>[^,]*)\,)(\s*)((t0=(?<t0>[^,]*)\,)?)(\s*)(trackId=(?<trackId>[a-zA-Z0-9]+)\})(\s*)(\[(?<time>[^]]*)\])(\s*)(\[(?<loglevel>DEBUG|INFO|WARN|ERROR)\])((.*\"time\":(?<apitime>\d+)\,\"code\":(?<apicode>\"[^\"]*\")\,\"msg\":(?<apimsg>\"[^\"]*)\"\})?)(.*\[Cost)?((\s+(?<Cost>\d+)ms\])?)" 
           } 
          
           } 
          
           } 
          
           mutate { 
          
           #convert => {"Cost" => "integer"} 
          
           convert => [ 
           "Cost" 
           , 
           "integer" 
           , 
           "request_time" 
           , 
           "integer" 
           , 
           "response" 
           , 
           "integer" 
           , 
           "upstream_response" 
           , 
           "integer" 
           ] 
          
           } 
          
           } 
          
           output { 
          
           elasticsearch { 
          
           hosts => [ 
           "192.168.2.184:9200" 
           , 
           "192.168.2.187:9200" 
           , 
           "192.168.2.189:9200" 
           ] 
          
           user => elastic 
          
           password => changeme 
          
           index =>  
           "logstash-%{type}-%{+YYYY.MM.dd}" 
          
           manage_template =>  
           true 
          
           flush_size => 50000 
          
           idle_flush_time => 10 
          
           } 
          
           }

啓動logstash

192.168.2.184

nohup /apps/svr/logstash/bin/logstash -f /apps/conf/logstash/logstash-kafka.conf &

192.168.2.187/192.168.2.189

nohup /apps/svr/logstash/bin/logstash -f /apps/conf/logstash/logstash-in-kafka.conf &

應用服務器的filebeat的配置

 
           cd /apps/svr 
          
           tar zxvf filebeat-5.2.2-linux-x86_64.tar.gz 
          
           ln -s filebeat-5.2.2-linux-x86_64 filebeat 
          
           [root@java1732 svr]# sed -n ‘/^[^#]/’p filebeat/filebeat.yml 
          
           filebeat.prospectors: 
          
           - input_type: log 
          
           paths: 
          
           - /var/log/messages 
          
           document_type: sys-messages 
          
           output.logstash: 
          
           # The Logstash hosts 
          
           hosts: [ 
           "192.168.2.187:5044" 
           , 
           "192.168.2.189:5044" 
           ] 
          
           #調試指令：./filebeat -e -c filebeat.yml -d 「production」 
          
           啓動： nohup ./filebeat -c filebeat.yml -e &

kibana頁面配置

 
           cd /apps/svr 
          
           tar zxvf kibana-5.2.2-linux-x86_64.tar.gz 
          
           ln -s kibana-5.2.2-linux-x86_64 kibana 
          
           [root@17161 kibana]# sed -n ‘/^[^#]/’p config/kibana.yml 
          
           server.port: 5601 
          
           server.host:  
           "192.168.2.189" 
          
           elasticsearch.url:  
           "http://192.168.2.189:9200" 
          
           kibana.index:  
           ".kibana"

啓動kibana

nohup bin/kibana &

#對應的nginx的配置以下

 
           upstream kibana { 
          
           keepalive      400; 
          
           server  192.168.2.184:5601 max_fails=3  fail_timeout=30s; 
          
           } 
          
           server  { 
          
           listen          80; 
          
           server_name     192.168.2.184; 
          
           if  
           (-d $request_filename) { 
          
           rewrite ^/(.*)([^/])$ http: 
           //$host/$1$2/ permanent; 
          
           } 
          
           location / { 
          
           proxy_pass              http: 
           //kibana; 
          
           proxy_http_version 1.1; 
          
           proxy_set_header Connection  
           "" 
           ; 
          
           proxy_set_header        X-Real-IP  $remote_addr; 
          
           proxy_set_header        Host             $host; 
          
           proxy_set_header        X-Forwarded-For  $proxy_add_x_forwarded_for; 
          
           } 
          
           error_log           logs/kinaba5.error.log; 
          
           access_log          logs/kinaba5.access.log log_access;

}

#至此，整個框架已經完成，能夠先創建kafka的topic測試，而後觀察elasticsearch的索引是否創建成功，或簡單的從頁面觀察便可

插件和其餘相關

一、因爲上述大部分應用都是跑在後臺，有時候進程是否掛掉，不得而知，監控若是對於每一個進程監控略顯麻煩，並且不方便啓動，因此這裏用supervisor進行統一管理，上述已經有安裝記錄，具體的配置就不作展現了二、 x-pack的安裝 /apps/svr/kibana/bin/kibana-plugin install x-pack